Twitter announced on Tuesday a significant change in the type of data it makes available free of charge to academic researchers interested in studying user behavior and trends related to online speech. Now, Twitter says it will no longer require researchers to pay for premium or corporate developer access and instead make the “complete history of public conversations” – what the company calls the full file search endpoint – available for any researcher or developer who applies as part of the launch of a new line of academic research.
The move is part of the company’s ongoing efforts to improve the Twitter API, the set of tools it makes available to outsiders to build something on Twitter or use its mountains of data for research studies. Twitter launched its new and improved API last summer as a way to make peace with the developer and research communities, which often conflict with the continued and sometimes adversarial development of the platform’s product over the years. This contentious relationship started mainly in 2012, when Twitter decided to cut off the main developer’s access to the platform to maintain tighter control over what data it would allow potential competitors to use.
Twitter recognizes the less than ideal relationship it has with the developer and the research communities. “Our developer platform has not always made it easier for researchers to access the data they need, and many have had to rely on their own resources to find the right information. Despite this, for more than a decade, academic researchers have been using Twitter data for discoveries and innovations that help make the world a better place, ”write product managers Adam Tornes and Leanne Trujillo in a blog post published today.
Tornes and Trujillo point out the steps Twitter took last year to improve its tools, including the launch of a free COVID-19 endpoint and a beta version of the new academic research trail to gather adequate feedback before today’s public launch. And the company is making a concerted effort to convince researchers that it wants to help them, especially now that social media discourse has become more crucial to understanding online misinformation, electoral interference, hate speech and other topics that have occupied the center stage during and after 2020 US elections.
In addition to opening its public archive, Twitter says it is also giving approved candidates a monthly tweet volume limit of more than 10 million tweets, which is 20 times higher than what was available in the standard free range before. It also allows for more precise filtering to help researchers find tweets and other data relevant to what they are studying and “new technical and methodological guides” to help researchers find what they are looking for and make better use of it in studies.
Twitter says interested academics and developers can sign up for the company’s new academic research line developer site. The track, however, will come with some limitations. For now, the company is already opening access to independent researchers or journalists. You will have to be a student or be part of an academic institution. Twitter also says it will not provide access to account data that has been suspended or banned, which can complicate efforts to study hate speech, misinformation and other types of conversations that violate Twitter rules.
It also means that the @realDonaldTrump account cannot be accessed via the Twitter archive either after Trump’s widespread depletion on U.S. social media following the deadly attack on the U.S. Capitol. Still, Trump’s tweets are archived online through his official presidential library, while independent websites and database tools like The Trump Twitter Archive and ProPublicaPolitwoops has also preserved the former president’s tweets, including those excluded.
“We heard a lot of interest from the academic research community in studying @realDonaldTrump,” Trujillo said during a call with reporters on Monday. “We are talking internally about how we can carefully consider studying this topic.”
January 26 update, 15:47 ET: Added more information on access to suspended or banned accounts and tweets from former President Donald Trump.