Silver Lake in town, as mentioned. There’s shoreline access with a little room to roam at the public beach.
Swan Lake just east of town is also a possibility. The road-based access points are more limited, but there is State Natural Area land that connects to the shoreline where you could hike in.
The Wisconsin River holds muskies right in town. The shoreline on Conant St. from Paquette Park through the Sunset Park landing is very easy to access. And west of town on the south side of the river Levee Rd through the Pine Island Wildlife Area offers literally miles of access.