Uncovering the Internet of terrible things (exclusive)

SAN FRANCISCO—What if there weren’t just a handful of illegal online drug marketplaces like “Silk Road” that exist on the shadowy fringes of the Internet, but rather hundreds and possibly thousands? New research for the first time accurately maps such hidden Internet back alleys. And the map resembles a constellation of interconnected stars.

Using a combination of human and computer analysis, British computer security intelligence company IntelliAgg has determined that the stable portion of the deliberately hidden Dark Web turns out to be smaller, but also far more heavily skewed toward illicit use, than previously thought. It is releasing a report (PDF) about its findings on Monday here at the annual computer security-focused RSA Conference.

Of the nearly 30,000 Dark Web sites the company mapped, it has so far analyzed the contents of more than 8,000 sites and discovered illegal content on 49 percent of them through a combination of automated computer and human analysis. Those sites were dedicated to marketplaces for trading child pornography, drugs, weapons, money laundering, computer-hacking services, and stolen financial data such as credit card numbers and corporate secrets.

By comparison, the Internet-at-large that you can access through a standard Web browser included an estimated 1 billion sites as of 2014.

The Dark Web “is like the Internet circa 1999. People had just started mapping it then,” said Emerson Tan, a technical adviser to IntelliAgg. “I can’t tell you who is looking at the biggest niche porn site, but now I can tell you what that site is.”

IntelliAgg’s tracking and mapping methodology has the potential to become a powerful new tool for law enforcement. Although the company cannot deanonymize people using the Dark Web, it can see which sites they’re visiting and the types of data they are exchanging. Instead of having to wait for a Dark Web site like Silk Road to get big enough to track because of its real-world impact, law enforcement agencies—and even executives worried about corporate espionage—can follow trails of data exchanges.

One of several sites IntelliAgg adviser Emerson Tan showed The Parallax that was accessible through the Dark Web and cataloged by its map. Screenshot by IntelliAgg.

IntelliAgg co-founder and Chief Technical Officer Thomas Olofsson says the company has been sharing data with law enforcement agencies in Europe and the United States, and it has lined up more than a dozen “big-name Fortune 500” corporate clients “who want to know their exposure to data leakage online.” Oloffson declined to name any of IntelliAgg’s corporate clients.

IntelliAgg’s report concludes that “about 68 percent” of the Dark Web content it has analyzed without computer assistance is “unlawful, according to U.S. and U.K. law.”

Law enforcement agencies in Europe and the United States did not respond to requests for comment for this story. In a review of hundreds of news stories about crimes committed with the assistance of the Dark Web, in fact, The Parallax did not find one in which a U.S. law enforcement agency directly commented on the services used. Even in court documents, agencies referred to the anonymizing software used to access the Dark Web most often in terms such as “an anonymizing network.”

How IntelliAgg maps the Dark Web

To begin mapping the Dark Web, IntelliAgg added hundreds of its own computer servers to the anonymizing network called The Onion Router (Tor) Project. The open-source Tor, the largest of the Dark Web networks available, is the one people most often refer to when discussing the Dark Web or Dark Net. (It is different from the Deep Web, which includes sites you can reach from any Web browser but that, for various reasons, are not cataloged by search engines such as Google and Bing.)

The U.S. Navy created Tor in 2002 as a smokescreen to protect people who live in countries without freedom-of-speech protections from being tracked when they connect to the Internet. With Tor’s services, people can use public-facing Internet services such as Facebook without being tracked by any organization except the host. They can also use Tor to create and run hidden online marketplaces for drugs or pornography. It’s like an Internet overlay.

“Hidden services enable people to talk and transfer files in a surveillance-resistant fashion. But they also allow you to host a website which has been completely hidden from the world,” Tan says.

The thickness of Tor’s smoke was thought of at one point as impenetrable, though in 2014 the FBI successfully identified people who were hiding behind Tor’s cloaking service. Tor is used by political dissidents, whistleblowers, human rights advocates, and journalists. Research by IntelliAgg and others also points to high percentages of illegal activity on the network; a 2015 Trend Micro report (PDF) found 26 percent of the Dark Web relates to child pornography and exploitation.

Employees of the Tor Project declined to comment for this story. However, Tor co-founder Roger Dingledine in a 2013 blog post described the two sides of the network:

“On the one hand, you have a girl in Syria who is alive right now because of Tor,” he wrote. “On the other hand, you have a girl in America who is harmed by some jerk, and the jerk uses Tor. So, how do you balance these two? How do you decide which one is more important, or more ‘valuable’ to the world? The answer is that it’s the wrong question to ask: you aren’t actually going to save the girl in America by getting rid of Tor. Whereas getting rid of Tor *would* harm the girl in Syria (along with a wide variety of people and groups around the world).”

To connect to Dark Web sites and protect your identity, you need Tor’s encryption software. It hides who you are by encoding the signals your computer normally transmits to the “regular” Internet. It then bounces your request to view the contents of the Dark Web site through multiple computer servers before reaching its destination and sending those contents back to you. Think of it as tubes within tubes, and each tube can’t see the layer inside it—or any other layer. These layers are why it’s called The Onion Router.

A weapons marketplace on the Dark Web cataloged by IntelliAgg’s map. Screenshot by IntelliAgg.

By having access to hundreds of those servers, at a cost of “tens of thousands of [British] pounds,” Olofsson says, IntelliAgg can view the hidden directory services of Dark Web sites on Tor. Without hidden directory services, it would be impossible to find any site on Tor.

IntelliAgg also uses private link lists and Web-crawling software to determine the contents of a Tor site. When a previously undocumented Tor site is discovered, IntelliAgg’s software sends out a signal to determine whether the site is available to unencrypted or encrypted requests. A Web browser emulator gathers data from the site, then feeds it to IntelliAgg’s database.

IntelliAgg’s machine-learning system, which the company unofficially calls the Auguries, “had achieved an accuracy of 94 percent, compared to an experienced human analyst” accuracy rate of 100 percent, according to IntelliAgg’s report.

Cleaning up the Dark Web

In addition to protecting people looking to remain anonymous on the Internet for legitimate reasons, Tan says Tor’s structure is resistant to common website disruption tactics, such as man-in-the-middle attacks and DNS spoofing.

Tor might be underutilized because of its reputation as one of the more horrible corners of the Internet. The FBI recently charged 137 people as part of a massive child pornography distribution ring who were hiding on Dark Web sites only accessible through Tor.

Tan says IntelliAgg has no interest in ruining Tor’s anonymization services for people engaged in political dissent.

IntelliAgg’s internal database of Dark Web sites. Screenshot by IntelliAgg.

“We can’t deanonymize people,” Tan says, referring to those using the Dark Web.

Olofsson concurs. “To deanonymize a person, you’d need half the exit nodes,” he says. “But we’re monitoring a large portion of the network. We can watch people request stuff inside the Dark Web.”

Law enforcement agencies armed with personal data, Tan says, could use IntelliAgg’s research to help it build a case against a suspected drug trafficker or child pornographer. With its new technology, IntelliAgg plans to help law enforcement agencies and corporations pinpoint illegal use of the Dark Web.

“People who use Tor for anonymity reasons and political reasons will get what they need,” Tan says, “But people who use Tor for criminal activity—watch out because the hiding portion of the game is essentially up.”