Buffalo Bore and Grizzly Cartridge Co. both make "heavy" .357 loads with 180 gr. hard cast FN bullets that would be the cat's meow for what you are planning.
You will be fine w the .357. I’ve carried mine in similar circumstances and had no worries.
The key as you indicated as well as others, is bullet selection. 180 gr hard cast hand loads or from one of the above mfgs is your huckleberry.
