Definitely! It does seem to be extra expense and to what gain? I assume that yours is the original chambering and not reamed and bored out? Like you said, you sure aren't going to have it rebarreled.
When I was considering my rebarrel I, too, debated between the .338-06 and .338-284 but wanted something just a little 'weirder' ('course ended up with the 6.5 in the end)! Ballistics differences being negligible, you can't go wrong with either. I wouldn't rule out a .338-284 in my future but, for now, my Whelen has got me covered!