The 6.5 Swed will work fine on Elk with the loads you stated. Elk are no harder to kill than any other animal if you shoot them in the right place. Bigger Cal's don't make up for poor shooting. Lung them and you have Elk back strap.
A Doe walks out of the woods today and says, that is the last time I'm going to do that for Two Bucks.
