I admit to a bias toward the .204, both because of the ballistics and because in my experience it does recoil less than the .223. I noticed that immediately on shooting my first one, and it has continued through the next three rifles.

Sierra recoil program results:

.204 Ruger--
40-grain bullet at 3700 fps
28 grains of powder
8.5 pound rifle:
4.0 foot-pounds

.223 Remington--
40-grain bullet at 3800 fps
28 grains of powder
8.5 pound rifle:
4.3 foot-pounds

“Montana seems to me to be what a small boy would think Texas is like from hearing Texans.”
John Steinbeck