This was was mirrored on Feb 6, 2013

RubyGems 1/30/13 Incident Status

Hi everyone! We’re using this Google Doc as a live way to update folks outside of our existing status site (also it keeps changes/history pretty well).

All of this is going on in #rubygems on irc.freenode.net (and several sub-rooms). If you want to help out, join in. RubyGems is a volunteer-run service.

For current status on what services are up, please check http://status.rubygems.org/ 

Gem downloads are available and every gem installable by `gem install` has been verified by one or more SHA512 checksums. No gems were altered during the exploit.

Need to push a new gem? Use Bundler’s git option until the Push API is back: http://gembundler.com/git.html 

1 February 2013

As of 2:05AM EST: (Eric Hodel)

  1. We’ve verified all but 13689 gems, all prerelease versions, with 2 or more third-party hashes. This verification reduces the likelihood of collusion between the person who exploited rubygems.org and one of the submitted sets of SHA512 checksums.
  2. See https://github.com/rubygems/rubygems-verification for data and scripts used in all rubygems verifications.

31 January 2013

As of 8:50PM EST: (Eric Hodel)

  1. We’re still working to set up a new rubygems.org host on AWS.  Currently we're tuning postgres, configuring loadbalancers and auditing access rules

As of 6:55PM EST: (Eric Hodel)

  1. The S3 logs have been checked and there have been zero duplicate PUT requests for gems, so no existing gem was overwritten by the exploit.
  2. The 80 prerelease gems that had not been verified have been yanked.
  3. All gems on S3 that are downloadable through `gem install` have been verified against at least one third-party mirror with SHA512 checksums.
  4. At some point in the future we will delete the remaining unverified but uninstallable gems from S3.

As of 6:40PM EST: (Eric Hodel)

  1. All but 80 prerelease gems have been verified. Except for these 80 gems the gems from the unverified list had been yanked and were not installable by `gem install` or through bundler. The gems in the unverified list posted below in the Notes section will be yanked and the authors of those gems will be notified by email.

As of 4:00PM EST: (Brad Landers)

  1. Clarification for rubygems users: the old rubygems.org infrastructure is not in use. The API is not actually online, even though you are still able to install gems. Workarounds are in place that allow this to happen, but the old infrastructure and app are offline. The actual gem files continue to be hosted on Amazon S3 (same as always).

As of 3:05PM EST: (Mark Imbriaco)

  1. Yesterday all gems were downloaded and checked for YAML exploits in their metadata and no odd “!ruby” payloads were found: https://gist.github.com/fd090905e322c44148b9

As of 2:45PM EST: (Eric Hodel)

  1. The unverified list from 2:15PM EST was incorrectly generated and excluded gems with a mismatched checksum from any of the mirrors.  This has been corrected.  The difference was 49 gems.

As of 2:35PM EST: (Eric Hodel for evan)

  1. Red Hat security teams helped with forensics analysis on the server:
  1. The server was well-patched
  2. No evidence of tampering with processes was found
  3. No evidence of shell access was found
  4. No evidence of running commands was found

As of 2:15PM EST: (Eric Hodel)

  1. Updated the unverified list at https://gist.github.com/4685276 to include terrcin’s Railscamp NZ mirror
  2. The list now has 5.8% of gems unverified by at least one mirror

As of 2:00PM EST: (Eric Hodel)

  1. This list of hashes has gems that have not been verified by at least one mirror: https://gist.github.com/4685276
  2. This list leaves 9.8% of gems unverified by at least one mirror

As of 12:40PM EST: (qrush)

  1. Waiting on west coast gem verifiers to wake up and confirm work done.
  2. Infrastructure team has started spinning up instances and testing out provisioning

As of 9:40AM EST: (qrush)

  1. We are still working to verify 100% of the gems.
  2. New infrastructure work has really come together at https://github.com/rubygems/rubygems-aws - hoping to deploy soon!

As of 3:15AM EST:(Eric Hodel/raggi)

  1. The Heroku checksum set (built from heroku application caches between late November 2012 and 21 January 2013 at midnight UTC containing 17496 checksums) verified an additional 710 gems that were not verified by the Automatic Labs or Blue Box sets. This is 0.2% of gems. Currently 90% of gems match S3 with two or more mirrors.

As of 2:00AM EST: (Eric Hodel/raggi)

  1. Automatic Labs mirror checksums match the Blue Box mirror checksums giving 90% coverage with two mirrors except for 50 gems in the Automatic Labs checksums

As of 1:10AM EST: (Eric Hodel/raggi)

  1. The gem with a mismatched checksum from the Blue Box has been checked and verified.

As of 12:00AM EST: (qrush)

  1. Infrastructure work is coming along nicely at https://github.com/rubygems/rubygems-aws 
  2. Some potential workarounds for the YAML vulnerability on rubygems.org’s Gem::Specification loading have been proposed and patches are coming together.

30 January 2013

As of 11:40PM EST: (Eric Hodel/raggi)

  1. Verified gems against the Blue Box mirror with 90% of gems matching, the remaining 10% are missing with the exception of one gem with a mismatch due to an incomplete cache. This list cross-checks with the AT&T list. The sha512 sum list here: https://gist.github.com/8db00eea60c8d99d2bfb

As of 11:30PM EST: (qrush)

  1. Status site is fixed, please check that for individual up/down status checks
  2. Individual gem pages are redirecting to static index.html site now

As of 11:00PM EST: (qrush)

  1. Status site is reporting incorrect information since the index.html was put in place. Looking into fixing it.

As of 10:30PM EST: (qrush)

  1. Chef & Verification work continues
  2. Looking into the exploit gem has started
  3. Maintenance page replaced with static index.html served off frontend boxes on Rackspace
  4. Individual gem pages are 404’ing right now (http://rubygems.org/gems/rails), looking into fixing this on frontends

As of 10:00PM EST: (qrush)

  1. Chef work continues
  2. More verification work continues
  3. Almost ready with index.html for frontend boxes
  4. From Heroku - “The RubyGems team has verified that about 84% of all gems stored at RubyGems are unmodified. An external 2-month old mirror was used for the verification, and the gap is due to gems pushed in the last two months. A second mirror is currently being used to cover the remaining 16%.”

As of 9:30PM EST: (qrush)

  1. Verification work continues with BlueBox and other friendly folks
  2. Working on getting an index.html page up on the frontend Rackspace boxes (that were not compromised), and then going to redirect
  3. Status site still blocked thanks to messed up Heroku deploy FIXED! Thanks @tmaher.
  4. 84% of gems verified against AT&T mirror which was last updated end of November 2012 (around the 26th)


As of 9:00PM EST: (qrush)

  1. Site is still pointed to Rackspace (but no requests are being served from the compromised box)
  2. Several BlueBox engineers have come to help with verifying gems. Thanks!!
  3. Status site can’t be updated easily on Heroku, so tweeted this doc out and going to keep using it.
  4. Google hangout has started for AWS move
  5. Fix is still pending for the original YAML vulnerability in the Rails app

As of 8:30PM EST: (qrush)

  1. The site is in maintenance mode.
  2. We need to spin down the compromised “backend” machine on Rackspace still.
  3. We are working on updating the status site since it’s been pretty broken
  4. A team is working on verifying all known gems on production.s3.rubygems.org to ensure none have been tampered with. Chances of this are extremely small (See below). Results will be published soon, hopefully.
  5. A team is working on setting up proper infrastructure with Chef on AWS to replace our existing hand-rolled infrastructure on Rackspace.

As of 6:30PM EST: (bradland)

  1. Rubygems team verify that S3 logs show no PUTs from hosts other than the production server or developer laptops; verification that S3 credentials were not used “in the wild”.

As of 3:00PM EST: (qrush)

  1. GitHub and Heroku recommend to stop deploys until all gems can be verified: https://twitter.com/heroku/status/296693401573797889

As of 12:00PM EST: (evan)

  1. Auditing and verification of gems begins in #rubygems on freenode

As of 11:00AM EST: (qrush/evan)

  1. Site brought back up without Push API

As of 9:45 AM EST:  (qrush/evan)

  1. Put site into maintenance mode
  2. S3 keys reset, SECRET_TOKEN reset
  3. Exploit gems deleted from database and purged from S3

As of 9:30 AM EST: (bradland)

  1. Link to “exploit” gem (that’s literally the name) posted to Hacker News

IRC channels on Freenode

#rubygems Central conversation

#rubygems-verification: Verifying checksums against S3

#rubygems-aws: Rebuilding the site on a new host

#rubygems-exploit: Patching rubygems-org app

How to help

Assistance is greatly appreciated. Most of the effort is being coordinated by an ongoing conversation in IRC #rubygems. Please take a moment to read the message backlog before flooding the channel. Currently, there is a lot of effort going on in verifying gems across many sources. You can help by:

  1. Validating your gems
  1. View this gist and grab one of the validators from the “Notes”, “Important links” section below
  2. Validate your local gems
  3. Comment on that gist with any discrepancies
  4. Join #rubygems-verification if you have data
  1. Contact someone on IRC in #rubygems-verification if you know of a gem mirror/cache with a significant (thousands or more) number of gems
  1. We already have Heroku, Blue Box and Rubyforge, thx!

What’s happening right now

  1. Rebuild of rubygems.org infrastructure (new home) on AWS us-west
  2. Verification of .gem files from as many sources as possible
  3. Assessment of core vulnerability

Official status info

  1. Canonical: http://status.rubygems.org 
  2. Twitter: https://twitter.com/rubygems_status 

Notes

Important links:

  1. See https://github.com/rubygems/rubygems-verification for a list of a list of SHA512 checksums and tools used to verify the gems on S3. This contains data and scripts mentioned below.
  1. Rubygems S3 .gem file list list: http://spy.infr.as/all-gems.txt
  1. Entire list has been grepped by benchMark (mark imbriaco) for !ruby invocations (clean)
  1. .gem file checksums
  1. Unverified list (SHA512): https://gist.github.com/4685276 This list is updated as new checksums are created. Check the Revisions for the last update
  2. RubyGems (MD5): http://cl.ly/MY8P
  3. RubyGems (SHA512): http://cl.ly/MYie
  4. RubyForge (MD5): https://gist.github.com/69c89387c6c3bfcf8719
  5. AT&T Mirror (SHA512): https://gist.github.com/4679063 
    does not include prereleases or the last 2 months
  6. Blue Box Mirror (MD5): https://gist.github.com/f9c4e5cf78aee261ad48
  7. Blue Box Mirror (SHA512): https://gist.github.com/8db00eea60c8d99d2bfb mismatch omg_validator-1.0.0.gem was separately validated as OK
  8. Automated Labs Mirror (SHA512): https://gist.github.com/4680849 contains 50 mismatches which also mismatch against the Blue Box checksums
  9. Heroku (SHA512) https://gist.github.com/4681280 from application caches built between late November 2012 and 21 January 2013, some gems have multiple hashes
  10. Mirror generated by terrcin for NZ Railscamp (SHA512) https://gist.github.com/4685502
  11. Google third-party depot (SHA512): https://gist.github.com/raw/c23588f19b7171247bb2/f963d04c31c078df45fd341ea15edf79def0449f/google3rd-shas
  12. Tokyo RubyGems Mirror (SHA512): https://gist.github.com/raw/d1a5316e4b8522e09e15/60ec496eca44ba139c3315452089eceaae7c0321/tkm-shas
  13. Google “ghost” (SHA512): https://gist.github.com/raw/f8cdb4d40197ee28670b/7cf4f14e79d97318198a7b3df1153d079c7b2aaf/ghost-shas
  1. Gem SHA comparison tool: https://gist.github.com/e4cb74f17d7a995d32e8

    This tool is used to compare the RubyGems SHA512 list above against the mirror SHA512 checksums below.  It’s comment describes it’s usage.

    We’ve been generating the unverified list by subtracting matching mirror SHA512 checksums from the RubyGems list so the unverified list contains all gems that have at least one matching checksum.
  2. Yanked gems were removed from the unverified list leaving 80 gems. These 80 gems are all prereleases and will be yanked. The authors will be emailed.

    Here is the tool and files used to remove yanked gems:     https://gist.github.com/d7e7820e6ee65d7254b2

    Here is the list of unverified gems:     https://gist.github.com/raw/d7e7820e6ee65d7254b2/c96af82860203ee3ac1dd93747e24fbc7c417f92/unverified_minus_yanked.txt
  3. Validators
  1. Gem validators (code examples): https://gist.github.com/4a4e005b575d71c92a24 
  2. Validate your local gems against S3: https://gist.github.com/4678189
  3. Validate your local gems against S3 (RVM compat): https://gist.github.com/4678250 
  4. Validate your local gems against S3 (rbenv compat): https://gist.github.com/4679488
  5. Validate a .gem file against your installed gems: https://gist.github.com/4678778
  1. Rubygems S3 .gem last modified dates (via curl HEAD): http://cl.ly/0L3M31022Q46
  2. Rubygems Versions#created_at data: https://gist.github.com/4677292
  3. Original exploit gem
  1. Exploit gem (original PoC): https://www.dropbox.com/s/fi7r5oovqyrtacm/exploit-gems.zip 
  1. This gem will not harm your system, but you should not install it (obviously)
  1. Exploit metadata: https://gist.github.com/75af5e0b071d247c08a8 
  2. Exploit gem payload: https://gist.github.com/3e4829f79dbd1be11295
  1. This gem uploaded rubygems.org *.yml data to pastie.org
  2. The attacker had the credentials contained therein

Discussion:

  1. HN discussion: http://news.ycombinator.com/item?id=5139583 
  2. Notes regarding exploit origin: http://news.ycombinator.com/item?id=5140100
  3. IRC at #rubygems (please try not to bombard the channel with questions; the issue is being actively worked)

What happened?

A user uploaded a malicious gem that contained a malicious gem manifest (YAML file). The manifest contained embedded Ruby with this payload. This is the only known incident involving this vulnerability, but the vulnerability involved is a remote code execution exploit, so the usual rules apply.

The Rubygems admins turned off the ability to push new gems (to prevent further) exploits of the same vuln, and immediately started efforts to download and examine every gem manifest to rule out additional exploit gems. None were found, and efforts continued as outlined in the rest of this document.

Posts about the incident

  1. http://blog.newrelic.com/2013/01/30/new-relic-and-rubygems-security/
  2. http://blog.librato.com/2013/01/rogue-ruby-gem-can-publish-librato.html

Incident response recommendations

Note: removed anticipated/current highlighting because of indications from the Rubygems team that an infrastructure rebuild is under way.

  1. Assume that the rubygems.org host is owned by a malicious entity
  1. Note that any work performed verifying existing gems is moot if the box is owned, and we may never know if the box is owned
  1. Consider all gems subject to tampering until the rubygems.org infrastructure has been replaced
  1. Can use S3 last modified to re-verify only gems changed since last verification
  1. Perform analysis to identify potentially altered gems
  1. Mirror cross-checking: run hashes against the .gem files on the Rubygems S3 bucket and cross-check them against other mirrors
  1. Some gem sources appear to have repacked when publishing gems, so some versions show a mismatched checksum, even though the contents are identical; workaround is to unpack gem and compare contents
  1. With the knowledge that S3 last modified time stamps cannot be altered (there is no S3 API to change them), compare to other data to identify discrepancies
  1. Rubygems data Versions#created_at should relate closely (seconds) to the S3 modified date data (qrush verified that uploads are not handled asynchronously)
  2. PUT log data from the app should verify against S3 last modified dates; discrepancies would indicate that an out-of-band update was made to files on S3