Yes, worth doing it. Wind and cold factor is when you want them, and they help without wind too. Keeps holes from freezing up, make the airspace above the hole sealed to the interior, big difference in temps that way, even if you bank snow. Do it well, and you really don’t have to bank.
I’ve gone from homemade hole sleeves in first skid house, to Fish Hole Buddies in first wheelhouse, to Catch Covers in my new wheel house. Sprung for the safety web covers, too, saved a cell phone last year, totally worth it, saved an errant foot/leg, too. They are easy to pull off quick before ripping lips. Also got the handle covers for summer camping to save the toes. Sweet system.
Buy the sleeves that match the brand of hole covers you have, and trim them to height on a driveway dropped down, and do exactly what MNdrifters says – just jam them down into the slush after drilling to create a good seal. I’d never go back to not having them.