According to the weapons database for the Yellowstone series, in the first season there were 2 lever actions use, Winchester 94 and a Marlin 336. This season I would imagine it's the same 336 as in the first season. Calibers were not listed.
I don't watch the series, I have seen a screen shot of Rip and a scoped Marlin but it doesn't show much of the gun other then the action and the scope.