We send out Special Reports whenever we are able to share one-time, high value datasets that we feel should be reported responsibly for maximum public benefit. Although the events included in these Special Reports fall outside of our usual 24 hour daily reporting window, we believe that there would still be significant benefit to our constituents in receiving and hopefully acting on the retrospective data.
In response to the major ongoing global security incident relating to Apache Log4j 2 that began on December 9th, we published a one off Special Report on Vulnerable Log4j Servers on December 15th 2021. That Special Report provided National CSIRTs and network owners with information about Internet facing systems detected on their networks that were potentially vulnerable to CVE-2021-44228. We also blogged initial observations from our honeypot sensor network relating to Log4j and attempted CVE-2021-44228 exploitation. We encourage you to read our previous posts for background, initial analysis and remediation information.
CVE-2021-44228: Update 2021-12-22
During the week since our first Special Report, security vulnerabilities in Log4j have continued to make headlines in the mainstream media as well as in the cyber security industry. Multiple Log4j 2 patches have been released by the Apache Foundation to address the original CVE-2021-44228 vulnerability, as well as other subsequently discovered additional vulnerabilities CVE-2021-45046 (remote code execution – RCE) and CVE-2021-45105 (Denial of Service – DoS) too. Software product/platform/service assessments and code reviews have discovered many impacted software packages, requiring large scale remediation activities world-wide. Internet-wide mass scanning and exploitation of the vulnerabilities on exposed systems has continued, including by alleged nation state actors and ransomware operators. The US Cybersecurity and Infrastructure Security Agency (CISA) has ordered all civilian Federal agencies to patch the critical Log4j vulnerability by December 24 and system defenders are still racing to patch the vulnerability in the run up to Christmas.
At the time of writing (December 22nd 2021 22:00 UTC) we have seen 174,068 scan/exploit events from 2,344 unique IP addresses across 433 ports. That compares with 38,638 scan/exploit events from 1,939 unique IP addresses across 272 ports at the time of our initial report (December 14th 2021 23:00 UTC). Scan/exploit events rates detected by our global honeypot sensor network peaked on December 20th and have since fallen slightly:
The top source countries observed in our original report (December 14th 2021 23:00 UTC) were:
Compared with the current top source countries at the time of writing (December 22nd 2021 22:00 UTC):
Combined for comparison:
There has been a significant shift in the distribution of top CVE-2021-44228 scan/exploiter locations, which is more marked at the Autonomous System Number (ASN) level.
The top source ASNs observed in our original report (December 14th 2021 23:00 UTC) were:
Compared with the current top source ASNs at the time of writing (December 22nd 2021 22:00 UTC):
Selectel in Russia has now become the top CVE-2021-44228 scan/exploit source observed by our honeypots.
The top TCP ports where CVE-2021-44228 scan/exploit attempts were observed by our honeypots to date were:
As per our
previous Special Report and own policies, Shadowserver are not ourselves scanning IPv4 /0 for exposed Log4j vulnerabilities. But Alpha Strike Labs, who kindly provided scan data used in our original Special Report, have completed a second major scan and have once again kindly offered to share their scan data with us in order to help more National CSIRTs and network owners be quickly notified about potentially vulnerable Log4j systems. This scan data was included in a second run of our Special Report that was sent to Shadowserver’s 132 National CSIRTs in 173 countries and territories and over 6000 network owners worldwide overnight on December 22nd. You can find more detail on the report format here. We would like to thank Alpha Strike Labs for once again sharing their data and helping the Internet community respond to this serious ongoing cyber security threat.
CVE-2021-44228 Data Analysis
The data provided for this second Special Report was collected by Alpha Strike Labs between December 16th and 22nd 2021. Their scan once again generated an outbound DNS response from vulnerable systems through the JNDI interface. The latest Special Report includes 10951 identified vulnerable Log4j instances on 8472 unique Internet facing IP addresses across 1500 ASNs in 116 countries (it is possible for an IP address to have an exposed log4j instance on more than one listening port). That compares to 13294 identified vulnerable Log4j instances on 13294 unique Internet facing IP addresses across 1602 ASNs in 109 countries in the original scan.
The ports and instance counts included in the second scan data set were:
The top HTTP headers that returned a response when queried were:
The IP-geolocated distribution of CVE-2021-44228 vulnerable IP addresses still features an unusual mix of countries, potentially showing how widely log4j is used internationally. The rate of remediation between individual countries is visibly different after one week.
Global distribution of vulnerable IP addresses per country in our original report (December 15th):
Global distribution of vulnerable IP addresses per country at the time of writing (December 22nd):
Distribution of vulnerable IP addresses per US county in our original report (December 15th):
Distribution of vulnerable IP addresses per US county at the time of writing (December 22nd):
Global distribution of vulnerable IP addresses per country in our original report (December 15th):
Global distribution of vulnerable IP addresses per country at the time of writing (December 22nd):
Global distribution of vulnerable IP addresses per country in our original report (December 14th):
Global distribution of vulnerable IP addresses per country at the time of writing (December 22nd):
Although starting from a higher number of exposed vulnerable Log4j servers, remediation efforts in countries such as the US, Ireland and Germany have reduced the number of vulnerable systems significantly, whereas Iran show only limited change in the number of systems detected by scanning. The number of detected systems in China, South Korea, Russia and Hong Kong has increased (although note that this could be due to more effective and complete scanning for the vulnerability).
For a more detailed IP-geolocated view of where detected vulnerable Log4j servers are located, click the small maps for high resolution world and regional images below.
Distribution of vulnerable IP addresses in our original report (December 15th) – Worldwide:
Distribution of vulnerable IP addresses at the time of writing (December 22nd) – Worldwide:
Distribution of vulnerable IP addresses in our original report (December 15th) – US:
Distribution of vulnerable IP addresses at the time of writing (December 22nd) – US:
Distribution of vulnerable IP addresses in our original report (December 15th) – North America:
Distribution of vulnerable IP addresses at the time of writing (December 22nd) – North America:
Distribution of vulnerable IP addresses in our original report (December 15th 2021 23:00 UTC) – Central America:
Distribution of vulnerable IP addresses at the time of writing (December 22nd 2021 22:00 UTC) – Central America:
Distribution of vulnerable IP addresses in our original report (December 15th) – South America:
Distribution of vulnerable IP addresses at the time of writing (December 22nd) – South America:
Distribution of vulnerable IP addresses in our original report (December 15th) – Europe:
Distribution of vulnerable IP addresses at the time of writing (December 22nd) – Europe:
Distribution of vulnerable IP addresses in our original report (December 15th) – Middle East:
Distribution of vulnerable IP addresses at the time of writing (December 22nd) – Middle East:
Distribution of vulnerable IP addresses in our original report (December 15th) – Africa:
Distribution of vulnerable IP addresses at the time of writing (December 22nd) – Africa:
Distribution of vulnerable IP addresses in our original report (December 15th) – China:
Distribution of vulnerable IP addresses at the time of writing (December 22nd) – China:
Distribution of vulnerable IP addresses in our original report (December 15th) – Oceania:
Distribution of vulnerable IP addresses at the time of writing (December 22nd) – Oceania:
Log4j Remediation
Technical details about the various vulnerabilities in log4j 2 and good remediation advice has been provided by a number of trusted sources:
https://logging.apache.org/log4j/2.x/
https://www.ncsc.gov.uk/news/apache-log4j-vulnerability
https://us-cert.cisa.gov/apache-log4j-vulnerability-guidance
https://www.cisa.gov/uscert/ncas/alerts/aa21-356a
https://www.govcert.ch/blog/zero-day-exploit-targeting-popular-java-library-log4j/
Apache Mitigations are available for version 2.10 and higher. Patches and mitigations and patches should be applied to all internet and internally facing log4j systems immediately since this vulnerability represents the maximum risk/impact possible and is being actively exploited at scale now.
Direct link to the Special Report format is here.
If you missed this Special Report because you were not yet a subscriber to our free daily network reports, do not worry: simply subscribe for your network or country now and specifically request all recent Shadowserver Special Reports. We will resend the Special Report specifically for your network or country (for National CERT/CSIRTs).
NB: We only send relevant reports for YOUR network/IP addresses – you will not get notifications for networks that you fail to request reports on, or that are not rightfully your responsibility.
If you have a data set which you feel could also be of benefit to National CERT/CSIRTs and network owners world-wide to help protect victims of cybercrime, please get in touch and discuss the options for using Shadowserver’s proven reporting systems for distribution and remediation.