The Jambos took the lead after six minutes thanks to a breakaway from inside their own half, involving Paul Hartley and Calum Elliot. The move culminated in Hartley sending the ball low across the face of goal when Edgaras Jankauskas side-footed into the net.
Two minutes later the Edinburgh side doubled their lead through captain Steven Pressley. The big defender was unmarked on the edge of the six-yard box when a corner came in from Rudi Skacel and he headed high into the goal.
Celtic were by no means out of the game and Stilian Petrov created several scoring chances, but the visitors could not find the back of the net.
The Hoops were lucky to finish the half with all 11 players after a thumping tackle from Bobo Balde on Deividas Cesnauskas just outside the penalty area. Balde escaped with a yellow card and the resulting free-kick failed to trouble Artur Boruc.
Celtic's afternoon looked set to go from bad to worse when their most dangerous player, Petrov, was forced off with a hamstring injury with less than five minutes of the second half gone.
But Petrov's replacement Stephen Pearson had only been on the field for five minutes when he scored to bring Celtic right back into the game.
The goal buoyed the Glasgow side and they began to exert more pressure on a tired looking Hearts.
Shunsuke Nakamura and Pearson both had chances to equalise, and when the hosts were reduced to ten men with the dismissal of Takis Fyssas for a challenge on Shaun Maloney, Celtic's chance of taking something from the game improved.
It came as no surprise when they equalised after 87 minutes. Nakamura took a free-kick just outside the area and Stephen McManus struck from close range.
Two minutes later McManus shattered Hearts' afternoon when he scored the winner.