Attack on Codecov Affects CustomersCompany Warns Clients' Information May Have Been Exfiltrated
Codecov, a company that tests software code prior to release, has notified customers that attackers had access to its network for a month and placed malware in one of its systems, which may have led to the exfiltration of customers' information.
The company says it learned from a customer on April 1 that attackers had gained access to its Docker image creation process and extracted the credentials needed to access and modify the company's Bash Uploader and other internal systems. An investigation determined the attackers routinely accessed the company's network for about a month.
"Our investigation has determined that beginning Jan. 31, 2021, there were periodic, unauthorized alterations of our Bash Uploader script by a third party, which enabled them to potentially export information stored in our users' continuous integration environments. This information was then sent to a third-party server outside of Codecov's infrastructure," the company says.
Codecov did not say how many customers may have been affected. Its website says more than 29,000 organizations, including Google, Palo Alto Networks and Procter & Gamble Co., use its tools.
The company notes Basher Uploader works with the company's Codecov-actions uploader for Github, the Codecov CircleCl orb and the Codecov Bitrise Step, so these were also affected by the attack.
Supply Chain Attack?
Bash Uploader is widely used and likely embedded in thousands of DevOps pipelines, says Setu Kulkarni, vice president for strategy at WhiteHat Security.
Quinn Wilton, senior researcher with Synopsys Software Integrity, calls the Codecov incident a supply chain attack along the lines of the SolarWinds attack, which affected 18,000 of its customers and led to follow-on attacks on nine government agencies and 100 companies.
"In both cases, we're seeing attackers leverage weaknesses in supply chain security, and this dynamic means what while it is the vendor that is being initially breached, the impact of that breach is felt by that vendor's customers," Wilton says.
A Codecov customer uncovered the issue on April 1, reporting a discrepancy between the shasum fetched from the company's Bash Uploader and one located on Github. A shasum is a script used to compute SHA message digests, according to Linux.
The company notes the altered Bash Uploader - a tool that gathers reports and uploads them to the Codecov environment - could potentially expose:
- Any credentials, tokens or keys that customers passed through their continuous integration runner that would be accessible when the Bash Uploader script was executed;
- Any services, datastores and application code that could be accessed with these credentials, tokens or keys;
- The git remote information (URL of the origin repository) of repositories using the Bash Uploaders to upload coverage to Codecov in their continuous integration.
Some Codecov customers took to Twitter to voice displeasure with the company's apparent two-week delay in reporting the attack.
Two weeks where that attacker had access to our credentials and we had no clue. Mistakes are acceptable. This delay is not.— Marco Cardoso (@macardoso95) April 15, 2021
But the company says it delayed the notification until it could obtain accurate information through its investigation.
Kevin Beaumont, a senior threat intelligence analyst at Microsoft, notes it will not be easy for those affected to defend their networks.
"Good luck to network defenders hunting, as they've withheld IoCs citing a law enforcement investigation (that's not a good reason for something like this)," Beaumont tweeted.
Customer Mitigation Recommendations
Codecov offered several suggestions to its customers for reinstituting a level of security for their information held by the company and on Github, Gitlab, Bitbucket or any third-party repository where code was stored.
"We strongly recommend affected users immediately re-roll all of their credentials, tokens or keys located in the environment variables in their CI [continuous integration] processes that used one of Codecov's Bash Uploaders," the company says.
Codecov says customers can check which keys and tokens in their continuous integration environment are in danger by running the "env" command in their continuous integration pipeline. If this action returns anything private or sensitive, the credential should be invalidated and replaced.
"Additionally, if you use a locally stored version of a Bash Uploader, you should check that version for the following: curl -sm 0.5 -d "$(git remote -v). If this appears anywhere in your locally stored Bash Uploader, you should immediately replace the bash files with the most recent version from https://codecov.io/bash," the company says.
Customers should take this action for any repository that used one of Codecov's Bash Uploaders from Jan. 31 to April 1, the company says.
Codecov says it's taken several actions to safeguard its environment, including:
- Rotating all relevant internal credentials, including the key used to facilitate the modification of the Bash Uploader;
- Auditing where and how the key was accessible;
- Setting up monitoring and auditing tools to ensure that this kind of unintended change cannot occur to the Bash Uploader again;
- Working with the hosting provider of the third-party server to ensure the malicious web server was properly decommissioned.