Trinity Attack Incident Part 1: Summary and next steps

IOTA Foundation ·

Update: The IOTA Trinity wallet is deprecated with the Chrysalis upgrade and will be replaced by the new Firefly wallet. You can easily transfer your tokens following these instructions.

Summary: Trinity is a software wallet for the IOTA digital asset that has been developed for desktop and mobile operating systems. Managed by the IOTA Foundation, this open-source software project enables the user to manage their tokens over the IOTA network. On February 12, 2020 the Trinity Wallet was attacked via a third-party dependency from Moonpay, which resulted in the theft of around 8.55 Ti in IOTA tokens.

This blog post is divided into a 3 part series:

  1. Part 1 summarizes the series of events that led to the attack and the measures taken by the IOTA Foundation. (This blog)
  2. Part 2 is the seed migration plan put in place to protect users that might have been affected by the attack. You can read it here.
  3. Part 3 offers an overview of key learnings, takeaways and measures that the IOTA Foundation will implement to ensure the highest security standards for all of our software development. You can read it here.

The following outlines the Trinity Attack Summary and measures taken by the IOTA Foundation to protect user’s tokens.

Series of Events

On Wednesday, 12th of February 2020, around 3 PM CET, moderators on the IOTA Discord server started receiving reports from users who were observing a zero balance and/or unauthorized outgoing transactions on their previously positive-balance accounts. It became clear this was not an isolated incident, and the IOTA Foundation’s engineering teams began to work on identifying the cause.

Within the first four hours of investigation, the Foundation made the decision to halt the coordinator, which was put in place as a temporary security mechanism during the network’s maturation phase. The decision to halt the coordinator is not one taken lightly, as it suspends the confirmation of value transactions on the network. Nonetheless, to prevent the attacker from transferring further tokens, it was an essential step. As a result, the attacker was unable to successfully obtain all targeted tokens, and a number of transfers were stopped en route to the attacker.  

In order to enhance transparency around this incident, the Foundation implemented a Major Incident Management plan, which included regular status updates via a dedicated website.  

This allowed us to provide public updates whenever possible, but also continue to work diligently behind the scenes to investigate different scenarios, including:

  1. A possible breach of the IOTA core protocol;
  2. Malicious modification of the available Trinity installer, across all or single OS versions;
  3. DNS hijacking, where a modified Trinity version would be downloaded from the hacker’s server;
  4. Virus/trojan infection resulting from phishing attacks;
  5. Remote code injection, e.g. through a dependency;
  6. Insecure seed generation (similar to the phishing incident from early 2019);
  7. Organized social-engineering attack (in relation to Binance’s recently announced 50x IOTA margin trading).

Early analysis and investigations (attack patterns, in-depth scans of affected users systems, extensive code dependency scans/reviews, different types of user-comparisons), as well as process of elimination, allowed the teams to identify a likely cause: the integration of a third-party service (Moonpay), which enabled users to directly purchase IOTA tokens within Trinity. We immediately informed MoonPay about the possible exploit.

At the time of its integration into Trinity, Moonpay was only available as bundled code delivered by a CDN (content delivery network), so the IOTA Foundation integrated it as such. Although widely used in web technologies, CDN delivery has inherent risks. One of those risks is that the code expected by the device could be unknowingly replaced with code that is not expected. The IOTA Foundation flagged the risks involved and requested an NPM (Node package manager) module to mitigate it. This was later published by Moonpay, after most of the integration work had already been done, but release pressure and human error added up to the Foundation not switching to the more secure NPM package prior to launch. This was the weakness leveraged by the attacker and one that could likely have been resolved if the Foundation had had a more extensive, cross-team review process for larger releases.

Over the course of the next 48 hours, the Foundation, with the support of a number of victims, collected information and obtained Trinity files from affected users. The Foundation’s internal analysis of affected Trinity caches found irrefutable proof that they had been compromised with one of several illicit versions of Moonpay’s software development kit (SDK), which was being loaded automatically from Moonpay’s servers (their content delivery network) when a user opened Trinity. The code was loaded into the local Trinity instance, and, after the user’s wallet was unlocked, decrypted the user’s seed and sent the seed and password to a server controlled by the attacker. Before transferring tokens out, the attacker awaited the release of a new Trinity version, which would overwrite Trinity’s cache files and thus remove the remaining traces of the hacker’s exploit. With this realization and code samples in hand, the IOTA Foundation immediately filed a report with the Berlin Police Cyber Division.

Through an attack analysis, performed by the IOTA Foundation, it became clear that the pattern of the attacker was consolidating multiple packs of 28 Gi. We suspect this value was chosen to keep the USD value of one pack under 10,000 USD and avoid triggering exchanges’ KYC identification procedures. We immediately contacted all exchanges with the results of the pattern analysis and asked them to lock associated exchange accounts. The first reply from nearly all exchanges was that they had not received any of the stolen token bundles. Due to the processing structure of the bundles, it was hard to shake the suspicion that the bundles had been sent to an exchange address. After escalating multiple times, we received sets of exchange deposit transaction logs. When we analyzed these logs with our Tangle analytics toolsets we, unfortunately, found that several addresses were owned by an exchange. We requested the exchange again to immediately lock the accounts, and are currently in further correspondence with them to assess the full picture of the amount of tokens the attacker was able to convert and transfer out of the exchange.

The next revelation came with the release of the log files to the IOTA Foundation on the 15th of February from the DNS provider contracted by Moonpay: Cloudflare. With the cooperation of Moonpay, we were able to get the logs of the past 18 months of their Cloudflare account. This, together with the security analysis, painted a very clear picture of the stages of an evolving attack that dates back to November 27th, 2019.

The Moonpay integration into Trinity officially began in September 2019, with the first closed beta being opened on November 11th, 2019. Through a leak in the testing period, on November 12th 2019, the upcoming integration into Trinity became well known within our community. The integration was made public on our open Github repo in the morning on the 26th of November.

The attacker started on November 27th, 2019 with a DNS-interception Proof of Concept that used a Cloudflare API key to rewrite the api.moonpay.io endpoints, capturing all data going to api.moonpay.io for potential analysis or exfiltration. Another longer-running Proof of Concept was evaluated by the attacker one month later, on December 22nd, 2019. On January 25th, 2020, the active attack on Trinity began, where the attacker started shipping illicit code via Moonpay’s DNS provider at Cloudflare.

Over the next two weeks, the attacker refined the malicious code and exfiltration techniques using code obfuscation and modification of the Moonpay API endpoints. Within this window of time, the IOTA seeds were stolen. The process of code iteration and seed theft continued until the 10th of February (although there are indications that malicious SDKs were served even until the 14th of February), at which point Moonpay became aware of illicit routes and took action to delete the API key, change login credentials and remove inactive users. Unfortunately, the IOTA Foundation was not informed of the unsanctioned API access until observing it for ourselves in the Cloudfare logs received from Moonpay on February 15th.

Without API access, the attacker was alerted to the fact that the route of attack was gone — and on the next day, the 11th of February, began executing transactions using the hijacked seeds. This theft was then ultimately interrupted when the coordinator was halted on the 12th of February. At present, the IOTA Foundation is aware of 50 independent seeds that had their tokens stolen during this attack, which amounts to a total of 8.55 Ti.

Trinity users will still need to use the forthcoming migration tool to protect their tokens from further thefts.  

The nature of this attack introduced several complexities for the IOTA network to successfully resume operations without causing further potential losses to token holders who have used the Trinity wallet. As such, the Foundation has taken the extra precautionary step to develop a detailed migration plan and a dedicated tool to protect users who might have been affected by this theft and offer all Trinity users a safe way to migrate their tokens to a new seed. The exact details of this migration plan will be shared with the community in a subsequent blog post (Part 2).

Steps Taken to Address the Incident

  • The Foundation set up as tatus update page where victims and the public could access regular updates.
  • Built a new Tangle analytics toolset (utilizing our permanode) that tracks tokens in real-time. This tool will help support the ongoing criminal investigation.
  • Allocated all available resources to assist with the investigation of attacked seeds and analyze the attack pattern, using the set of newly developed tools, as well as a separate parallel manual analysis and verification (to validate tooling reliability).
  • Released a new version of Trinity Desktop for users to install on top of the current version with the attack vector removed, which would allow users to safely open and check their wallet balances. You can find it here.
  • Released new versions of Trinity Mobile on iOS and Android with MoonPay removed. These can be downloaded via the App Store and Play Store respectively.
  • Developed an attack remediation plan, which involves building a seed migration tool to move users to a safe seed.
  • Brought on multiple security experts and firms to assist with the analysis and cyberforensic investigation, as well as develop the remediation plan.
  • Contacted the UK, German, and Maltese police and the FBI to report the incident and provided documentation and updates as they became available.
  • Collected information from affected users and developed a dedicated community discord channel for them.
  • Collected and analyzed app files from both affected and non-affected users, categorized malicious code types and developed a timeline of when the malicious code was deployed.
  • Contacted all relevant exchanges to gather insight into where the tokens had been transferred and to lock any unsold tokens.
  • Worked together with MoonPay to investigate the cause of this hack and acquire the necessary information for the investigation.

Message to our community and users

We want to thank our extremely supportive community for offering their assistance during this crucial time period. We realize that having tokens stolen is a very stressful and emotional time for those affected, which is why we take this incident very seriously. We’ve made a lot of progress in getting to the bottom of this attack in a short time period and our engineers have been working diligently with law enforcement to analyze all events leading up to the attack and identify the culprit. We appreciate the patience of our community and users as we develop and implement tools that will assist in the recovery of stolen tokens.

Conclusion

Due to the ongoing cooperation and investigation by law enforcement and external security contractors, we are still analyzing specific details and events of the theft, and as such are not yet able to provide the community the complete portrayal of the incident. Hopefully, over the coming weeks and in cooperation with the involved parties, we will be able to provide everyone with detailed insight into the way in which these events unfolded.

Although some might say that a wallet-hack is a rite of passage in the crypto-industry, this in no way reduces the disappointment that the people in the IOTA Foundation feel for not meeting the standards we have set for ourselves. We fell short in fully-vetting the Trinity wallet continuously after new integrations, and apologize for letting our community down. We are currently working on a remediation plan for victims that had their tokens stolen by the malicious actor and continue to be in direct contact with them. We aim to publicly communicate a concrete plan next week. Separately from this plan, the Foundation continues to be in contact with the involved exchanges and law enforcement to hopefully find the perpetrator and recover as many of the stolen tokens as possible.

Key learnings, takeaways and measures for the Foundation’s development and security procedures will be shared in part 3 of this blog post series. We hope that the positive outcomes of this incident (namely, improved and tighter security procedures) will not only help to improve IOTA’s development but will also benefit the broader DLT ecosystem.  

Please continue to Part 2 of this series for more details on the Attack Incident and Migration Plan.