Hour Zero: A Post-Mortem
At 11:30 A.M. we announced our alpha release. Two hours later, we discovered a bug.
- The bug prevents purchasing Primes from the Pool.
- Current Prime holders are not at risk due to this bug. If you have Prime tokens, you can still freely exercise. We recommend you do so.
- No funds are at risk due to this bug.
- We paused the Pool, preventing new liquidity from being added.
Lines 235–241 are the area of focus here. In line 237, the premium is calculated. If this premium is 0, the transaction will revert with the error: “ERR_BAL_ETH” at line 241. We made the mistake of assuming the premium should only return 0 if the
amount is 0, which is the input parameter for the amount of Primes to buy. However,
premium is returning 0 due to the volatility being calculated and returning 0.
- 1:41 P.M.: Notified of an error being thrown when a user attempts to purchase Prime options from the pool.
- 2:05 P.M.: Cycled the active liquidity in the Pool. This means we took all the tokens out of the Pool, and re-added liquidity. We had no issues with withdrawing every single token we originally deposited!
- 3:19 P.M.: Identified and replicated the error in a test environment. Originally we thought the issue was a front-end one, but further investigation showed us the root cause, which was in the PrimePool contract.
- 3:26 P.M.: Paused the pool. While no funds were at immediate risk due to this bug, we paused the pool to prevent new liquidity being added to the pool. This was so that we can deploy a new Pool contract without affecting any additional addresses beyond our own.
We made the mistake of assuming the calculated premium in line 236 should not return 0.
In line 293, we make no extra checks to assure that the premium variable (which is returned) is greater than 0. This is a mistake.
We removed the if conditional statement which would revert the transaction if the premium is 0. You can see that if we submit a buy order with an
amount parameter that is not 0 (#1: in the event image below), we still get a 0 premium paid. We can see that in the buy event below, which we tested on the forked mainnet.
Another mistake we made is that our testing suite did not have coverage of this! In our tests, we make a lot of individual buy orders rather than consecutive buy orders. This means that the bug was never being thrown, because only one buy order was going into each new contract instance at a time. We would have caught the bug if we submitted two buy orders back to back.
We suspect the 0 value being returned is due to the division in one of the calculations. If the numerator is smaller than the denominator, it will return 0 because there are only integers in solidity land. There are three calculations that go consecutively: calculate the pool’s utilization, use the utilization in calculating the volatility proxy amount, set the global volatility variable to the calculated amount, then calculate the premium. In the premium calculation, the volatility is an input, and we multiply it. If the volatility is 0, the premium is 0. The volatility is indeed returning 0 in the case that Primes were bought from the pool one time. This makes all consecutive buy orders have a 0 value calculated for their premium, which in turn reverts the transaction.
New Pool Contract
Over the next few hours, we will be pushing a fix that adds additional checks to these values so that they will only be 0 in the correct situations. The premium should be 0 if the amount sent into the contract is 0.
We will be redeploying only the Pool contract, which affects only Pool liquidity providers. Primitive is the only liquidity provider that is affected. We will still be able to withdraw our provided liquidity once the user holding 200 Primes exercises their options.
What do I do if I have Prime tokens?
Current Prime holders do not need to take any action. If you are holding Prime tokens, you can freely exercise your right to purchase 200 DAI with 1 ETH. This is because the Prime contract is completely separate from the pool contract. Since they are separate, it makes it simple to redeploy the pool contract with the fixed code.
Learning and Onward
We’ve learned that our testing suite is not the catch-all assertion that the contracts function in the intended ways. We failed to adequately check every line of code. Going forward, we will be covering our math in the contracts more diligently with its own test suite to validate the invariants.