I'm thinking you have a short in the line somewhere. If nothing is touching the wire, the battery should last for many weeks without being recharged. The charger pulls almost no power until the fence is discharged. Wet grass can pull a lot juice from a battery.

I have a small fencer that I use with my llamas when hunting. It runs on 4 D cells. I can run it a week and the batteries will still show full power.

“In a time of deceit telling the truth is a revolutionary act.”
― George Orwell

It's not over when you lose. It's over when you quit.