Online abuse: there's just so, so much of it. Social networks teem with harassment and trolling, so much so that companies have outsourced the work of content moderation to an army of laborers, typically overseas, often at an enormous mental and emotional toll to the workers themselves.
But what if you didn't need humans to identify when online abuse was happening? If a computer was smart enough to spot cyberbullying as it happened, maybe it could be halted faster, without the emotional and financial costs that come with humans doing the job. At SRI International, the Silicon Valley incubator where Apple's Siri digital assistant was born, researchers believe they've developed algorithms that come close to doing just that.
"Social networks are overwhelmed with these kinds of problems, and human curators can’t manage the load," says Norman Winarsky, president of SRI Ventures. But SRI is developing an artificial intelligence with a deep understanding of how people communicate online that he says can help.
A few years ago, Winarsky was at TED, the high-profile ideas conference, where he noticed more than a few discussions taking place around bullying. The folks at SRI started talking, and the idea for a venture started percolating. Then, nine months ago, a social network approached the SRI and said it had a major problem with bullying on its platform. The company, which Winarsky declined to identify, had already gathered a wealth of reports and data sets on bullying and offered them to SRI to see if its researchers could do anything to help curb the problem.
Siri, the voice recognition tech acquired by Apple for its iPhones, is probably SRI's most recognizable achievement. But SRI has also seen smaller successes, including the creation of Redwood Robots, which was acquired by Google. Some of its spin-offs, including Intuitive Surgical (robot surgeons) and Nuance (natural language software), have gone public.
In other words, understanding the ways people communicate is an area where SRI already has a wealth of experience. The company, Winarsky says, also already owned some pretty well-developed artificial intelligence algorithms that could, among other things, grade the quality of essays in standardized tests. The team realized they could apply those algorithms against the bullying data they received from the social network. “The results were more successful than we thought they’d be given the complexity of the problem,” he says.
It worked so well, in fact, that companies even began to ask whether the technology could be used to comb social networks for other nefarious activity, such as terrorist recruiting. "I think there’s going to be a whole new category of things SRI could be exploring," Winarsky says. He says he can imagine a whole new field emerging around such technology in the near future, calling it "cyberpersonal security."
Building the anti-bullying tech took about 250 people looking at the gigabytes of data that the social network provided, says Bill Mark, president of SRI's information and computing sciences group. The data included instances of bullying that human curators had already identified, Mark says. Researchers fed that data into machine learning systems that began to learn and duplicate, under supervision, the job of human content moderation.
Still, bullying can be subtle and complex. It's a tough thing to teach a computer to analyze. "Bullying is a difficult problem to solve because you have to understand context," Mark says.
As Winarsky explains, an AI system has to understand that harassment isn't limited to glaring red flags like swear words. It needs to be able to parse meaning. A bully could post a seemingly innocuous message, he says, such as, "Why don't you stay at home now and forever?" At other times, Winarsky says, a post really could be an innocent joke. SRI's technology is designed to understand the meaning behind a post and consider those differences. It also seeks to identify attacks by analyzing patterns of actions, the same way a cybersecurity algorithm might recognize the nature of an attack on a server.
The goal, Winarsky says, is to ultimately build a semi-automated system that would be able to detect with a high rate of accuracy what a human curator would flag. Using a combination of natural language processing—an AI discipline that seeks to understand the meaning behind language, parsing grammar and sentence models and structure—in addition to machine learning and statistical approaches, the system flags possible instances of bullying and gauges the likelihood that abuse is taking place. As the algorithm improves, Winarsky says, it should eventually lessen the load to be handed off to the human curators who have to manually determine if something is bullying or not.
An initial working prototype of the system could be ready for real world applications six to twelve months from now, Winarsky says, though a polished version could take longer. Right now, he says, the team is exploring spinning off the venture, or licensing the technology out to social networks---if they need it. Asked whether he thinks SRI’s effort is ahead of everyone else’s, Winarsky hedges. “It’s extremely hard to say,” he says.
“My guess is every major social network on the planet is working on similar efforts. We have to be modest. But we at least know that when it comes to machine learning algorithms, our experience runs deep.”
The question, of course, is whether SRI’s technology can overcome the tactics employed by smart abusers. Jamia Wilson, executive director of Women Action Media, a group Twitter appointed last fall to look at reports of harassment on the social network, says her main concern is that abusers are well-aware of the initiatives to curb harassment on networks—and employ sophisticated techniques to avoid detection.
Some of the problems Wilson saw while looking at Twitter included tactics like the “tweet and delete,” when harassers erase disparaging tweets not long after sending them, and “false flagging,” when trolls make fake claims about harassment to overwhelm the resources of the social network. “I think the system sounds interesting, but I doubt it would solve the problem completely,” Wilson says. “The complexity needs to be addressed, and it’s important that there would still be some moderation processes.”
WAM investigated 811 reports, 161 of which it forwarded to Twitter. But the effort was arduous, and involved looking at each manually submitted report, and at times following up with the individuals sending in those reports.
After WAM’s effort concluded, the report the organization put out was fairly disheartening---trolls often found it too easy to game the system. But Wilson says the group fully intends to engage in a second trial, and she would be absolutely interested in trying out SRI’s tool when the time came. She says, "We need as many different kinds of tools as we can get to tackle this problem."
