I've shot a few whitetail does with a 223 and 60gr. Partitions. Shot them right behind the shoulder with very good results. There is some pretty stout bullets in the .224 line up, but I don't think that there are any in the 204 that would be suitable for deer. As others have said it's all about the bullet.
