If speed is your thing then go with the 6.
Same bullet weight, same case, the bigger bore will shoot faster. I think your logic is backward there.
Same weight bullet in a 6 vs 6.5 ?
I'm not sure I understand your question.
Same weight bullet in both. As in a 95 grain bullet loaded in both a 6 CM and in a 6.5 CM. The 6.5 has a larger bore, so it more efficiently pushes the bullet down the bore. It won't matter much, but various data points to about a 50 fps velocity difference with 95 grain bullets, and the difference is more like 100 fps when you get to the 107. I'm sure if there were 120-123gr bullets in the 6, the difference would be larger still.
Since the OP alluded to deer hunting with a lightweight rifle, I'm guessing that lighter varmint bullets in the 6 are likely off the table.