Bigger doesn't matter, until it does.
If you can fire the bigger round as accurately as the smaller round then the larger round simply gives you a larger margin of error.
No matter how good the hunter, for a variety of reasons bullets end up landing slightly off the aiming point. If you're shooting a bullet that produces a 2" dia wound channel vs a 4" dia wound channel, the bullet that produces the 4" dia wound channel can be an inch further out from the aiming point and still do damage to the vitals.
If you're convinced you'll never place your shots around the edges, then a .223 or a .243 is all you need for NA game. If however you wish to have a bit of reserve terminal performance for those times that things don't go quite as planned, bigger is better.
