{ "id": "2c0ab649-d7c4-4cc9-9765-3e4fd84dc73c", "created_at": "2026-04-06T00:18:42.622363Z", "updated_at": "2026-04-10T03:24:29.127614Z", "deleted_at": null, "sha1_hash": "b8312b980315627421fee6d2be238786123a1d6c", "title": "Scanning VirusTotal's firehose | Sky Blueteam", "llm_title": "", "authors": "", "file_creation_date": "0001-01-01T00:00:00Z", "file_modification_date": "0001-01-01T00:00:00Z", "file_size": 626930, "plain_text": "Scanning VirusTotal's firehose | Sky Blueteam\r\nPublished: 2021-09-21 · Archived: 2026-04-05 19:50:27 UTC\r\nSep 21, 2021 · 584 words · 3 minute read\r\nLet’s say one of your adversaries is known for using a given malware family, custom or off-the shelf. Even if the\r\ncoverage is biased and limited, samples on VirusTotal (VT) are the low-hanging fruits that keep on giving.\r\nAt $WORK, we are lucky to have access to the Virus Total feeds/file API. This API endpoint is the firehose of\r\nVirusTotal: it allows downloading each sample submitted to VT in pseudo-real-time. The feed is unfiltered (we are\r\nnot talking about VT’s LiveHunt feature) so the volume is HUGE.\r\nWe set the crazy objective to extract and push IOC in real-time for a given malware family submitted to\r\nVirusTotal. For this blog post, as an example, we will focus on Cobalt Strike.\r\nThe steps are:\r\n1. Download each sample submitted\r\n2. Apply Yara rules matching the malware families we are interested in\r\n3. Automatically extract C2 configuration\r\n4. Disseminate IOC\r\nInitially, we used our on-premises infrastructure with 2-3 servers. Quickly, the operational maintenance killed us:\r\nOur Celery cluster was regularly KO.\r\nEverything had to be very carefully tuned (memory limits, batch size, timeout, retries), we were constantly\r\njuggling with the balance between completeness, stability, and speed.\r\nAdding an under-performing Yara rule could break the platform.\r\nIt was also not a good use of our computing resources as VT’s activity is not evenly spread across the day:\r\nour servers were under-used most of the day while overloaded during the peaks.\r\nGoing Serverless 🔗\r\nTaking a step back, it jumped out at us that this was a textbook example for a Serverless architecture. It was easy\r\nto refactor our on-prem code into self-contained functions and glue them together with Amazon SQS:\r\nhttps://skyblue.team/posts/scanning-virustotal-firehose/\r\nPage 1 of 3\n\nThe platform has been running smoothly for 18 months, and from an operational point of view, we love it:\r\nThe scalability of the platform allowed us to not mind anymore about the performance of each rule: we can\r\nadd our Yara rules quite freely instead of cherry-picking and evaluating carefully each addition.\r\nSQS handles the whole retry mechanism.\r\nAdding a new dissector is as easy as plugging a new Lambda function to the Amazon Simple Notification\r\nService (SNS) topic.\r\nEverything is decoupled, it is easy to update one part without touching the rest.\r\nEach new release of libyara increases its performance and it is directly correlated to the execution\r\nduration’s average.\r\nEverything is instrumented, we learned to love the AWS Monitoring Console.\r\nPerformance Stats 🔗\r\nFor those who like numbers, here is a screenshot of the activity of the last 6 months:\r\nOn average:\r\nA batch of samples is scanned in less than 30s\r\nhttps://skyblue.team/posts/scanning-virustotal-firehose/\r\nPage 2 of 3\n\nThere are always 45 Lambda functions running at any given time\r\n97% of the executions are successful\r\nWe send 150 samples per minute (before deduplication) to dissectors\r\nCobaltStrike 🔗\r\nWe are using CobaltStrikeParser from Sentinel One to parse the beacons, then we are sending the JSON output to\r\nour Splunk instance.\r\nThere are two uses of this data:\r\nThreat Hunting: tracking some Threat Actors\r\nProactive protection: adding proactively the IOC to a watchlist in our scope\r\nFor Threat Hunting perspectives, we implement alerting for things like:\r\nSpecific watermark identifiers\r\nPatterns in the C2 domain\r\nNon-standard values for some fields\r\nUse of some options or specific malleable profile\r\nRegarding proactive Defense, there is currently no automatic pipeline to push the IOC into a WatchList/DenyList\r\nfor one reason: it is not uncommon to see trolling BEACONs using legitimate and “assumed safe” domains. To\r\nmitigate that, we plan to have a kind of Slack/Mattermost bot that will make us approve each entry seamlessly.\r\nSource: https://skyblue.team/posts/scanning-virustotal-firehose/\r\nhttps://skyblue.team/posts/scanning-virustotal-firehose/\r\nPage 3 of 3", "extraction_quality": 1, "language": "EN", "sources": [ "Malpedia" ], "references": [ "https://skyblue.team/posts/scanning-virustotal-firehose/" ], "report_names": [ "scanning-virustotal-firehose" ], "threat_actors": [ { "id": "610a7295-3139-4f34-8cec-b3da40add480", "created_at": "2023-01-06T13:46:38.608142Z", "updated_at": "2026-04-10T02:00:03.03764Z", "deleted_at": null, "main_name": "Cobalt", "aliases": [ "Cobalt Group", "Cobalt Gang", "GOLD KINGSWOOD", "COBALT SPIDER", "G0080", "Mule Libra" ], "source_name": "MISPGALAXY:Cobalt", "tools": [], "source_id": "MISPGALAXY", "reports": null }, { "id": "aa73cd6a-868c-4ae4-a5b2-7cb2c5ad1e9d", "created_at": "2022-10-25T16:07:24.139848Z", "updated_at": "2026-04-10T02:00:04.878798Z", "deleted_at": null, "main_name": "Safe", "aliases": [], "source_name": "ETDA:Safe", "tools": [ "DebugView", "LZ77", "OpenDoc", "SafeDisk", "TypeConfig", "UPXShell", "UsbDoc", "UsbExe" ], "source_id": "ETDA", "reports": null } ], "ts_created_at": 1775434722, "ts_updated_at": 1775791469, "ts_creation_date": 0, "ts_modification_date": 0, "files": { "pdf": "https://archive.orkl.eu/b8312b980315627421fee6d2be238786123a1d6c.pdf", "text": "https://archive.orkl.eu/b8312b980315627421fee6d2be238786123a1d6c.txt", "img": "https://archive.orkl.eu/b8312b980315627421fee6d2be238786123a1d6c.jpg" } }