After a 50-minute delay because of a power failure, the home team went in ahead at half-time thanks to Shaun Maloney's penalty.
Bobo Balde looked to have cost his team two points when he carelessly lost possession in the box, with Paul Dalglish taking full advantage to equalise for the visitors.
However, just three minutes from full-time a Shunsuke Nakamura cut in from the right before shooting low under Livi keeper Roddy McKenzie.
In the early minutes of the match Maloney might have had a penalty and John Hartson mis-hit his shot when well placed.
Dalglish threatened briefly when he skipped past Stephen McManus, but Artur Boruc did well to block the striker's shot.
Maloney was unlucky when his powerful strike rebounded off McKenzie's foot before he saw his header drift narrowly wide.
Stilian Petrov shot straight at the keeper before Celtic's pressure finally paid off.
At the end of a flowing move Maloney collected Aiden McGeady's pass and as he advanced for goal, Greg Strong brought down the striker. Maloney despatched his spot-kick into the top corner.
A shocking mistake by Balde let Livingston back into the match at the start of the second half. The centre-back tried to beat Dalglish in his own penalty box but lost possession to the striker who slotted home from 15 yards.
Young substitute Michael McGlinchey passed up a wonderful chance to put Celtic ahead before Nakamura came to the home side's rescue.
Paul Telfer's pass found the Japanese playmaker wide on the right and he cut inside two defenders before unleashing a left-foot shot beyond McKenzie.