Wonder if you would need a faster twist rate to stabilize those heavier bullets?
Absolutely, my two .204s won't even stabilize the 40 gr. Hornady, let alone anything longer. They are marginal, at best, with the 40 Nosler. Both are factory barrels, of a NOMINAL 1-12" twist, but are probably more like 1-12.5", due to tolerances in barrel making machinery.
I don't think anyone makes a decent "deer bullet" in .20,currently, and don't expect to see one soon, either.