The Urban Attitudes Lab at Tufts University has conducted research on accessing "big data" on social networking sites for civic purposes, according to Justin Hollander, associate professor in the Department of Urban and Environmental Policy and Planning at Tufts.
About six months ago, Hollander began researching new ways of accessing how people think about the places they live, work and play. "We're looking to see how tapping into social media data to understand attitudes and opinions can benefit both urban planning and public policy," he said.
Harnessing natural comments -- there are about one billion tweets per day -- could help governments learn what people are saying and feeling, said Hollander. And while formal types of data can be used as proxies for how happy people are, people openly share their sentiments on social networking sites.
Twitter and other social media sites can also provide information in an unobtrusive way. "The idea is that we can capture a potentially more valid and reliable view [of people's] opinions about the world," he said. As an inexact science, social science relies on a wide range of data sources to inform research, including surveys, interviews and focus groups; but people respond to being the subject of study, possibly affecting outcomes, Hollander said.
Hollander is also interested in extracting data from social sites because it can be done on a 24/7 basis, which means not having to wait for government to administer surveys, like the Decennial Census. Information from Twitter can also be connected to place; Hollander has approximated that about 10 percent of all tweets are geotagged to location.
In its first study earlier this year, the lab looked at using big data to learn about people's sentiments and civic interests in New Bedford, Mass., comparing Twitter messages with the city's published meeting minutes.
To extract tweets over a six-week period from February to April, researchers used the lab's own software to capture 122,186 tweets geotagged within the city that also had words pertaining to the New Bedford area. Hollander said anyone can get API information from Twitter to also mine data from an area as small as a neighborhood containing a couple hundred houses.
For its first study, the Urban Attitudes lab at Tufts University created its own software to isolate and analyze tweets geotagged in the area of New Bedford, Mass.
Researchers used IBM's SPSS Modeler software, comparing this to custom-designed software, to leverage a sentiment dictionary of nearly 3,000 words, assigning a sentiment score to each phrase -- ranging from -5 for awful feelings to +5 for feelings of elation. The lab did this for the Twitter messages, and found that about 7 percent were positive versus 5.5 percent negative, and correspondingly in the minutes, 1.7 percent were positive and .7 percent negative. In total, about 11,000 messages contained sentiments.
The lab also used NVivo qualitative software to analyze 24 key words in a one-year sample of the city's meeting minutes. By searching for the same words in Twitter posts, the researchers found that "school," "health," "safety," "parks," "field" and "children" were used frequently across both mediums.
Comparing tweets with the minutes, the lab's preliminary findings presented at a Chicago conference on big data suggest that slightly more sentiment was expressed in the Twitter realm, and that people in both the formal and informal contexts had expressed similar interests.
Hollander said other researchers or applications could expand the timeline for extracting data from social media sites, to cast a wider net. There is also the possibility of analyzing the data for discussions, retweets, and people's positions on issues, although working with large datasets can be challenging, he said.
Maria Pina-Rocha, director of Management Information Systems for New Bedford, said there's a municipal use for the study's sentiment-combing software, especially for New Bedford's tourism business. Yet Pina-Rocha said she has reservations about whether Twitter is the right forum to assess opinions on city-related issues, and to what extent opinions shared there are done so for other reasons, such as a way to blow off steam.
"I would look at the data and it would play a role, but I'm not sure what weight I would give it," Pina-Rocha said. "I think it is worthwhile to consider."
Still, Pina-Rocha said, social networking sites are the future public forums. "That's the way we're going these days -- people have their opinions and they are putting it out there."
Using the study's way of categorizing sentiments on Twitter, Hollander said governments can mine the data to answer specific questions, such as: What is the overall feeling of a proposed municipal bike path? In this way, supplementing traditional information-gathering methods, he said, a city could get a sense of people's feelings about a project.
Still, there are limits to this type of data, Hollander said, because the sample is not representative of the wider population. For instance, younger users are more active on these sites, according to Philip Chester, who serves as town planner for Lebanon, Conn., who sticks mainly to e-mail. Even so, he acknowledged that a public hearing, which is one way he gleans public opinion, is also not a valid statistical representation of the population, and that there might be some benefit to the technology.
"It would organize what people are saying and it might be a tool for planners and other government officials to get some of the pulse of what's out there," said Chester, who studied alongside Hollander in graduate school.
Next up for the lab is a new study contrasting Twitter posts from four Massachusetts cities with the recent election results.
As for using the technology to help government, Hollander said that is part of the lab's mission. Hollander is now engaging in discussions with possible government partners, and he is also looking at opportunities with Foursquare, a location-based social media platform. He said is interested in studying other social platforms, including Facebook, Tumblr, Wanelo and Yik Yak.
"The idea of really mining the attitudes and opinions of people using the Internet, using social media, is really the future," Hollander said. "As government professionals, we really have to find ways to tap into this knowledge base and not just rely on the old fashion ways to understand how things are going."
