Belgian substitute Dieter van Tornhout scored the only goal of the game when he headed in Lee Johnson's cross six minutes from time.
Celtic had a stoppage-time penalty appeal turned down as Killie, for whom keeper Cammy Bell made several fine saves, held on to their lead.
"It's amazing, I feel very emotional at the moment, brilliant, fantastic," Bell told the BBC. "It just shows how much commitment the boys had, we did it for the manager (Kenny Shiels).
"Everybody dreams but we are realistic. I think the boys were fantastic.
"I think we deserved to win, I had a few saves to make but you are going to have to against Celtic."
Source: AFP
Source: AFP