Open Zeppelin's ERC20Votes extension is safe and self-contained. It has been audited and is currently deployed in thousands of token contracts.
The main tradeoff is gas usage. Fortunately, users can make that decision themselves.
Users can choose to activate their voting power by calling delegate. When an account is delegated, transfers will also update voting power. That update uses extra gas. Users who do a lot of transfers can always undelegate their tokens to save gas by delegating to the zero address.