What you have should be fine, especially with the puffy pants. I was out on the NM/CO border this past weekend where it was swinging between low 20s and mid 50s and was getting by with a pair of lightweight long underwear underneath a pair of gridded fleece Cabelas pants and I was fine.
