X-Force uncovers global NetScaler Gateway credential harvesting campaign

This post was made possible through the contributions of Bastien Lardy and Ruben Castillo.
这篇文章是通过巴斯蒂安·拉迪和鲁本·卡斯蒂略的贡献而实现的。

In September of 2023, X-Force uncovered a campaign where attackers were exploiting the vulnerability identified in CVE-2023-3519 to attack unpatched NetScaler Gateways to insert a malicious script into the HTML content of the authentication web page to capture user credentials. The campaign is another example of increased interest from cyber criminals in credentials. The 2023 X-Force cloud threat report found that 67% of cloud-related incident response engagements were associated with the use of stolen credentials.
2023 年 9 月，X-Force 发现了一个活动，攻击者利用 CVE-2023-3519 中识别的漏洞攻击未修补的 NetScaler 网关，将恶意脚本插入身份验证网页的 HTML 内容中以捕获用户凭据。该活动是网络犯罪分子对凭据兴趣增加的另一个例子。2023 年 X-Force 云威胁报告发现，67% 的云相关事件响应参与与使用被盗凭据有关。

In response to the widespread exploitation of CVE-2023-3519 CISA released an advisory document containing guidance on detection, incident response, mitigations and validating security controls. However, through multiple incident response investigations, X-Force discovered a new exploitation artifact related to CVE2-2023-3519 and developed additional guidance to be used in conjunction with CISA’s detection and response recommendations.
为了应对CVE-2023-3519的广泛利用，CISA发布了一份咨询文件，其中包含有关检测，事件响应，缓解和验证安全控制的指南。但是，通过多次事件响应调查，X-Force 发现了与 CVE2-2023-3519 相关的新漏洞利用工件，并制定了与 CISA 的检测和响应建议结合使用的其他指南。

This post will cover the initial incident that led to uncovering the larger campaign, the credential harvesting campaign, as well as the new artifact, considerations and recommendations for responding to remediating an incident involving CVE-2023-3519.
这篇文章将介绍导致发现更大活动的初始事件、凭据收集活动，以及响应修正涉及 CVE-2023-3519 的事件的新项目、注意事项和建议。

Incident overview  事件概述

X-Force identified the campaign through an incident response engagement where a client had discovered the script after investigating reports of slow authentications on the NetScaler device. The script which is appended to the legitimate “index.html” file loads an additional remote JavaScript file that attaches a function to the “Log On” element in the VPN authentication page that collects the username and password information and sends it to a remote server during authentication.
X-Force 通过事件响应活动识别了该活动，在该活动中，客户在调查 NetScaler 设备上身份验证缓慢的报告后发现了该脚本。附加到合法“index.html”文件的脚本加载一个额外的远程 JavaScript 文件，该文件将函数附加到 VPN 身份验证页面中的“登录”元素，该元素收集用户名和密码信息并在身份验证期间将其发送到远程服务器。

As part of the initial exploit chain, the attackers sent a web request to “/gwtest/formssso? event=start&target=” triggering the memory corruption documented in CVE-2023-3519 to write a simple PHP web shell to /netscaler/ns_gui/vpn. With interactive access established through the PHP web shell, the attacker retrieved the contents of the “ns.conf” file on the device and appended custom HTML code to “index.html” which references a remote JavaScript file hosted on attacker-controlled infrastructure.
作为初始漏洞利用链的一部分，攻击者向“/gwtest/formssso？event=start&target=“触发 CVE-2023-3519 中记录的内存损坏，将一个简单的 PHP Web shell 写入 /netscaler/ns_gui/vpn。通过 PHP Web shell 建立交互式访问，攻击者检索了设备上“ns.conf”文件的内容，并将自定义 HTML 代码附加到“index.html”，该代码引用了托管在攻击者控制的基础设施上的远程 JavaScript 文件。

To facilitate the credential harvesting, the JavaScript code appended to “index.html” retrieves and executes additional JavaScript code that attaches a custom function to the “Log_On” element which collects the form data containing the username and password information and sends it to a remote host through a HTTP POST method upon authentication.
为了便于凭证收集，附加到“index.html”的 JavaScript 代码检索并执行额外的 JavaScript 代码，该代码将自定义函数附加到“Log_On”元素，该元素收集包含用户名和密码信息的表单数据，并在身份验证时通过 HTTP POST 方法将其发送到远程主机。

Larger credential harvesting campaign
更大规模的凭据收集活动

From that initial engagement, X-Force identified multiple domains created by the threat actor – jscloud[.]ink, jscloud[.]live, jscloud[.]biz, jscdn[.]biz, and cloudjs[.]live – registered on August 5th, 6th and 14th, and leveraging Cloudflare to mask where the domains were hosted. After identifying the threat actor’s C2, X-Force was able to identify almost 600 unique victim IP addresses hosting modified NetScaler Gateway login pages, with concentrations in the United States and Europe. The earliest modification time stamp X-Force has identified for NetScaler Gateway login pages is on August 11^th, 2023, although the campaign could have begun closer to when the domains were registered.
从最初的接触开始，X-Force 确定了由威胁参与者创建的多个域 – jscloud[。Ink， JSCLOUD[.]Live， jscloud[.]Biz， JSCDN [.]Biz和CloudJS[.]实时 – 在 8 月 5 日、6 日和 14 日注册，并利用 Cloudflare 来掩盖域的托管位置。在识别威胁参与者的 C2 后，X-Force 能够识别出近 600 个唯一的受害者 IP 地址，这些地址托管了修改后的 NetScaler 网关登录页面，这些地址集中在美国和欧洲。X-Force 为 NetScaler 网关登录页面确定的最早修改时间戳是 2023 ^th 年 8 月 11 日，尽管该活动可能更接近域注册时开始。

Despite variations in the URL, all domains hosted an almost identical JavaScript file, with the only difference being the C2 domain listed in the file, and all captured credentials were sent to the same URI “/items/accounts/“. X-Force has also observed in some instances the threat actor appending the same URL, or a URL using one of the other domains, to the same victim login page, indicating this threat actor is likely opportunistically compromising vulnerable NetScaler Gateways.
尽管 URL 存在差异，但所有域都托管了一个几乎相同的 JavaScript 文件，唯一的区别是文件中列出的 C2 域，并且所有捕获的凭据都发送到相同的 URI“/items/accounts/”。X-Force 还观察到在某些情况下，威胁参与者将相同的 URL 或使用其他域之一的 URL 附加到同一受害者登录页面，这表明该威胁参与者可能机会性地破坏易受攻击的 NetScaler 网关。

While public reporting has highlighted how various threat actors, including suspected Chinese threat actors and FIN8, have exploited these vulnerabilities, X-Force has not observed follow-on activity and is unable to attribute this campaign at this time.
虽然公开报告强调了各种威胁行为者（包括可疑的中国威胁行为者和FIN8）如何利用这些漏洞，但X-Force尚未观察到后续活动，目前无法确定此活动的原因。

CVE-2023-3519 detection and investigation guidance
CVE-2023-3519 检测和调查指南

Considerations for evidence collection from NetScaler device:
从 NetScaler 设备收集证据的注意事项：

In the default configuration, the NetScaler device will rewrite the “ns_gui” folder upon boot resulting in the directory being destroyed during shutdown. Throughout various investigations, X-Force has recovered valuable evidence from the “ns_gui” directory including samples of web shells and modified versions of “index.html”. Organizations should be careful not to shut down the device prior to collecting an image or other evidence.
在默认配置中，NetScaler 设备将在启动时重写“ns_gui”文件夹，从而导致目录在关闭期间被销毁。通过各种调查，X-Force从“ns_gui”目录中恢复了有价值的证据，包括Web shell样本和“index.html”的修改版本。组织应注意不要在收集图像或其他证据之前关闭设备。

New artifact for CVE-2023-3519 detection: NetScaler application crash logs
CVE-2023-3519 检测的新项目：NetScaler 应用程序崩溃日志

Through X-Force incident response engagements involving CVE-2023-3519, X-Force analysts identified that the NetScaler Packet Processing Engine (NSPPE) crash files can contain evidence of the exploitation of the vulnerability. The crash files are located with “/var/core/<number>/NSPPE*”. Similar to the default log files on the NetScaler device, the crash files are stored in “.gz” archives so they will need to be extracted prior to analysis.
通过涉及 CVE-2023-3519 的 X-Force 事件响应参与，X-Force 分析师发现 NetScaler 数据包处理引擎 （NSPPE） 崩溃文件可能包含利用该漏洞的证据。崩溃文件位于“/var/core//NSPPE*”中。与 NetScaler 设备上的默认日志文件类似，崩溃文件存储在“.gz”存档中，因此需要在分析之前提取它们。

Crash file path example: /var/core/6/NSPPE-01-9502.gz
崩溃文件路径示例：/var/core/6/NSPPE-01-9502.gz

The crash files are not human readable by default however X-Force discovered that the crash files do contain string data that can be extracted using strings, PowerShell or any other tool that can print the strings of printable characters in files.
默认情况下，崩溃文件不是人类可读的，但是X-Force发现崩溃文件确实包含字符串数据，可以使用字符串，PowerShell或任何其他可以打印文件中可打印字符字符串的工具提取。

X-Forced observed that the NSPPE crash file timestamps aligned with the filesystem timestamps of the PHP web shells created through exploitation. In other instances, X-Force was able to recover commands being passed to the web shells as part of post-exploitation activities.
X-Force观察到NSPPE崩溃文件时间戳与通过利用创建的PHP Web shell的文件系统时间戳一致。在其他情况下，X-Force 能够恢复作为开发后活动的一部分传递给 Web shell 的命令。

Note on NetScaler log backups:
有关 NetScaler 日志备份的说明：

X-Force has observed that the default NetScaler audit configuration is to leverage circular logging and retain the last 25 log files with a maximum size of 100 Kb. When logs are rolled, NetScaler will retain older log files in “.gz” archives. X-Force has observed that some of the available CVE-2023-3519 detection tools available on the Internet, do not consider the log data within the “.gz” archives. Organizations should ensure to extract the log files from the archives or leverage a tool such as “zgrep” which can search within compressed files.
X-Force 观察到，默认的 NetScaler 审核配置是利用循环日志记录并保留最后 25 个日志文件，最大大小为 100 Kb。滚动日志时，NetScaler 会将较旧的日志文件保留在“.gz”存档中。X-Force 观察到，互联网上可用的一些可用的 CVE-2023-3519 检测工具不考虑“.gz”存档中的日志数据。组织应确保从存档中提取日志文件，或利用可以在压缩文件中搜索的“zgrep”等工具。

Considerations for detection strategies within NetScaler access logs:
NetScaler 访问日志中检测策略的注意事项：

X-Force recreated the exploit for CVE-2023-3519 by sending a GET request to “https://<VulnerableGateway>/gwtest/formssso?event=start&target=” however X-Force was not able to recover a record of any of the web requests associated with exploitation attempts in the access logs. It is not clear whether the lack of a log entry for the connection to the “formssso” endpoint is due to a configuration issue on the X-Force test instance or if the “formssso” does not log connections by design.
X-Force 通过向“https:///gwtest/formssso？event=start&target=”发送 GET 请求来重新创建 CVE-2023-3519 漏洞，但 X-Force 无法恢复与访问日志中的利用尝试相关的任何 Web 请求的记录。目前尚不清楚缺少与“formssso”端点连接的日志条目是由于X-Force测试实例上的配置问题，还是“formssso”没有按设计记录连接。

X-Force recommends that clients analyze the following log sources for evidence of post-exploitation activity in the following files with a particular focus on identifying entries indicative of interacting with a web shell:
X-Force 建议客户端分析以下日志源，以查找以下文件中的利用后活动的证据，特别关注识别指示与 Web shell 交互的条目：

• /var/log/httpaccess.log
• /var/log/httperror.log
• /var/log/httpaccess-vpn.log

X-Force recommends that organizations assess their access logs for POST/GET requests and anomalous PHP files. Examples of post-exploitation interactions with a PHP web shell observed by X-Force:
X-Force建议组织评估其访问日志中的POST/GET请求和异常PHP文件。X-Force 观察到的与 PHP Web shell 的利用后交互示例：

While during exploitation tests, X-Force was not able to recover the details of the commands executed via the PHP web shells recovered from incident response engagements with the access logs, X-Force still recommends organizations assess their NetScaler access logs for evidence of command execution in the event different web shells were used.
虽然在利用测试期间，X-Force 无法恢复通过 PHP Web shell 执行的命令的详细信息，这些命令是从事件响应参与中恢复的访问日志，但 X-Force 仍然建议组织评估其 NetScaler 访问日志，以便在使用不同的 Web shell 时执行命令的证据。

Considerations for detection strategies within NetScaler command history logs:
NetScaler 命令历史记录日志中检测策略的注意事项：

The CISA advisory recommends organizations assess bash.log and sh.log for evidence of malicious activity leveraging the following keywords:
CISA建议组织评估bash.log和sh.log，以寻找利用以下关键字的恶意活动的证据：

• database.php 数据库.php
• ns_gui/vpn
• /flash/nsconfig/keys/updated
• LDAPTLS_REQCERT
• ldapsearch
• openssl + salt 开片+盐

In addition to the CISA advisory, X-Force recommends organizations also consider assessing “/var/log/notice.log”, “/var/log/bash.log” and” /var/log/sh.log” (including the associated rollover “.gz” archives) for signs of post-exploitation activity using the following additional keywords:
除了CISA公告外，X-Force还建议组织还考虑使用以下附加关键字评估“/var/log/notice.log”，“/var/log/bash.log”和“/var/log/sh.log”（包括相关的滚动“.gz”档案）以寻找开发后活动的迹象：

• Whoami 哇
• base64 –decode base64 – 解码
• /flash/Nsconfig/keys
• &>> index.html &>>索引.html
• echo <?php echo <？php
• echo <script 回显<脚本
• /nsconfig/ns.conf

It is important for organizations to analyze command history logs in the correct context of the attacker’s operations. Evidence gathered from command history within the context of an attack involving CVE-2023-3519 will be focused on post-exploitation activity. Organizations should analyze process execution data sources (including command history logs on the device) for commands associated with reconnaissance, credential harvesting, lateral movement and downloading/uploading of data and not restrict their assessment to just what is provided within the keywords.
组织必须在攻击者操作的正确上下文中分析命令历史记录日志。在涉及 CVE-2023-3519 的攻击背景下，从命令历史记录中收集的证据将集中在利用后活动上。组织应分析与侦测、凭据收集、横向移动和数据下载/上传相关的命令的流程执行数据源（包括设备上的命令历史记录日志），而不是将其评估限制在关键字中提供的内容。

Considerations for remediation:
修正注意事项：

As noted in the CISA advisory, attackers were observed viewing NetScaler configuration files /flash/nsconfig/keys/updated/* and /nsconfig/ns.conf which “contain an encrypted password that can be decrypted by the key stored on the ADC appliance”.
正如CISA公告中所述，攻击者被观察到查看NetScaler配置文件/flash/nsconfig/keys/updated/*和/nsconfig/ns.conf，这些文件“包含一个加密的密码，可以通过存储在ADC设备上的密钥解密”。

X-Force also noted that there were multiple credentials and certificates stored in the NetScaler configuration files so organizations should consider changing certificates as well as all passwords as part of incident remediation.
X-Force还指出，NetScaler配置文件中存储了多个凭据和证书，因此组织应考虑更改证书以及所有密码，作为事件补救的一部分。

Indicators  指标

References 引用

• https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-201a
• https://www.mandiant.com/resources/blog/citrix-zero-day-espionage
• https://twitter.com/SophosXOps/status/1695143572272738790

原文始发于John Dwyer  Richard Emerson：X-Force uncovers global NetScaler Gateway credential harvesting campaign