If Tower Rook can single-handedly keep two people at bay, not to mention three, it's a fair bet those people wouldn't do too well anyway. Armor stacking or just double-teaming him (Rook himself, not his towers) should force him back easily...unless he has adequate support, in which case you can chalk it up to good teamwork.
The towers already have a number of disadvantages--
-They auto-target. So no focused fire on DGs when creeps or minions are around, or multiple DGs.
-They're static. Regulus, Spit UB to an extent, TB -- anyone with ranged attacks -- can hit them from relative safety. And minion swarms pretty much are ranged attacks.
-They're weak. The HP doesn't go up with levels nearly as much as most abilities. But that doesn't really matter too much because...
-Setting up a tower farm takes loads of mana. Attacking Rook himself should force him away because he won't be stacking HP or armor, giving more time to take down / push back the towers because he won't be able to stay at the front setting new ones up. If he does stack HP/armor then attacking the towers becomes more viable because he won't be able to keep it up and still use his roll or slam.
-Setting up a tower farm takes a minute and twenty seconds, assuming he can do so constantly without running out of mana, and disregarding the time it takes to take a couple steps to stop whacking creeps and set one up. Hitting early or destroying the farm takes it out of action for quite some time.