The Lithuanian was only on the pitch for three minutes when he drilled the ball home after 77 minutes and then produced a stunning strike four minutes from time to kill the tie.
Celtic had the better chances over the 90 minutes but Hearts looked as if they needed the win more after making an indifferent start to the season.
Hearts, after losing 5-0 on their previous visit, stamped their authority on the game from the outset as they troubled Celtic in the first 20 minutes.
The Edinburgh side had plenty of possession but, for all their endeavours, Artur Boruc was not put under any pressure in the first half.
Michael Stewart curled an early free-kick high and wide and Calum Elliot whipped a shot across goal but Celtic looked comfortable at the back.
Aiden McGeady had a great break for Celtic when he raced from deep in his own half but his final shot was wayward.
Scott McDonald worked himself an opening with a clever piece of skill but directed his left-foot shot straight at Anthony Basso.
There was a let-off when Hearts skipper Christophe Berra headed the ball rather too heavily back to his goalkeeper but Basso had his wits about him and stretched to save.
The goalkeeper did well shortly before half-time to tip a fierce shot from Scott Brown over his crossbar.
Celtic made a half-time change with Lee Naylor, who had taken a knock, going off to be replaced by Evander Sno.
Within a minute of the restart, McDonald almost broke the deadlock with a 16-yard shot but Basso was alert to it and managed to get his boot to the ball.
Sno missed a chance after a great cross from Scott Brown but he could not keep it down and another opportunity was lost.
Andrew Driver had a chance at the other end after a searching pass from Stewart but he rushed his effort and it finished well wide.
But Celtic always looked the more likely and Basso saved well again to keep out a 20-yard shot from Massimo Donati.
Sno should have done much better after being set up by Brown but he blasted the ball carelessly over the top.
But Hearts snatched the lead against the run of play when Driver looked to have taken the ball out of play before crossing for Velicka to apply a superb finish with his right foot.
He scored an even better goal with four minutes left when he ran across the Celtic defence before thumping an unstoppable 20-yard shot past Boruc.