A bug in Taraxa's on-chain DPoS contract was discovered on January 19, 2024 and fixed on the same day. Due to the bug, the Taraxa development team requested that Kucoin pause trading, deposits and withdrawals as a cautionary measure. Trading on Kucoin has since been restored as of January 24, 2024.
We compiled this report to transparently inform the Taraxa community of the incident, its resolution, and future measures to prevent this from happening again.
Timeline of the Incident
On January 19, 2024, the Taraxa development team ("we") was notified by a community member that there's a bug in our on-chain DPoS contract which allowed them to repeatedly claim validator commissions beyond what they're entitled to from the DPoS contract. The community member offered to return the coins, and we offered a portion as a bug bounty.
Due to the seriousness of the problem, we notified exchanges of the issue and requested that they pause deposits, withdrawals, and trading as a cautionary measure. Kucoin was the first to respond. At this time technical investigation was just beginning and we didn't know the cause or scale of the problem.
On the same day, we fixed the problem, published an upgrade, and put in place a preventative measure before the upgrade is activated on the network. Since the problem is under control, we rescinded our request to the other exchanges, and requested that Kucoin restore trading, deposits, and withdrawals. We submitted a technical bug report to Kucoin at their request.
By January 24, 2024, Kucoin's team had completed their review of our report and found it satisfactory, and trading was resumed.
The Bug and its Fixes
The bug in the DPoS contract can happen when the following conditions are true. A validator node has,
- participated in consensus
- unclaimed commissions
- been fully un-delegated (i.e., all of its stake un-delegated)
- been waiting for the un-delegation delay (~32 days) to elapse
When these conditions are met, the validator can repeatedly claim its unclaimed commissions over & over again. This is due to an oversight in the code whereby the node technically still exists (as un-delegation hasn't fully gone through yet) but the commission claiming logic doesn't reset because that part of the code thinks it doesn't exist anymore. Here's one example of such a transaction.
As of January 24, 2024, this bug has been fixed, the node upgrade published, and it has taken effect on the mainnet.
Prior to the upgrade taking effect, we also put in place a preventative measure. Because the bug can only happen when it has been fully un-delegated, we put in place an automated script coupled with a wallet that scanned the network, identified nodes that were fully un-delegated, and then delegate 1,000 TARA (minimum delegation amount) to that node, preventing the exploit. Here's the github repo for the prevention script.
Returning the Funds to the DPoS Contract
A total of 17.1 million TARA was overdrawn from the DPoS contract and the full amount has been restored.
Here's a breakdown,
- Of the 17.1 million TARA overdrawn,
- 14.4 million TARA were returned by the community member to this wallet.
- 2.7 million remaining TARA were awarded to the community member as a bug bounty
- Returning the 17.1 million TARA to the DPoS contract,
- 14.4 million will come from the funds returned by the community member
- 2.7 million will come from the Community & Ecosystem Fund as it covers bug bounties.
- Total of 17.1 million TARA returned to the DPoS contract.
Restoring Withdrawals & Deposits on Kucoin
As of this writing on January 26, 2024, 1PM PST, withdrawals and deposits of TARA are still suspended on Kucoin. Kucoin's technical team has asked Taraxa's development team to confirm that a sufficient number of validators on the network has upgraded.
As of January 26, 2024, 1PM PST, 93% of all stake (1.98 billion TARA out of 2.13 billion TARA) are currently delegated to validator nodes that have upgraded. Since this is well above the 66% + 1 TARA threshold necessary for the asynchronous PBFT process to progress, we feel the security hurdle has been met.
We have reported this data to the Kucoin team and we fully expect a speedy restoration of withdrawals and deposits just like trading.
Going Forward: Improving Technical Execution and Communications
We are undertaking two categories of efforts to minimize this from happening in the future.
On the technical execution front, we will integrate fuzzing into our continuous delivery process to improve edge case exploit detection, and set up scanner scripts to look for abnormal transaction behaviors that could alert us to a potential exploit in progress.
On the communications front, we have put in place external & internal communication policies to improve how we communicate with our partners, our community, and the team internally.
We want to thank our community for believing in us, Kucoin for being a highly reliable and professional partner, the engineers on the Taraxa team who figured out and fixed the bug in record time, and finally to the community member who not only alerted us of the bug but voluntarily returned the funds.
We will continue to strive to improve our operations to better serve the Taraxa ecosystem.
Stay tuned! 🪁