Both can make a big difference.

With an '06 sized case, brass can make a 2 grain difference it what's safe and what's not.

I've seen primers make 100 fps difference. I've also seen time where it makes no statistical difference.

So, it depends. Until you know what your doing, it's best to start low, work up, and build your data set.

You didn't use logic or reason to get into this opinion, I cannot use logic or reason to get you out of it.

You cannot over estimate the unimportance of nearly everything. John Maxwell