It was another important scalp for the newly-promoted surprise package from League One after winning at Derby on the opening day.
There had been little to choose between the sides up to the goal as both teams produced some excellent football although there had been little penetration into the penalty box.
But the goal brought the game to life and James Coppinger had a chance to add a second in the 39th minute but chipped over the crossbar after James Hayter had robbed Guillaume Beuzelin of possession.
Coventry stepped up the pressure just before the interval to earn a series of corners that caused the home fans some anxious moments.
Visiting skipper Scott Dann had a header cleared off the line by Brian Stock and then Neil Sullivan did well to claw another effort from Elliott Ward from under the crossbar.
Both sides continued to attack in the second half but Clinton Morrison miscued when well-placed to equalise in the 57th minute after Jay Tabb had created the chance.
Lewis Guy squandered a good opportunity when he fired wildly wide after Gareth Taylor had sent him clean through in the 62nd minute and missed another chance following good work by Coppinger a couple of minutes later.
Rovers should have won by a wider margin but James Hayter missed by inches in the 72nd minute and then Taylor headed wide following good work by James Chambers.
The home side finished on top with Adam Lockwood wasting another opportunity when he headed wide with the goal at his mercy seven minutes from time.