Andy Ritchie accused his FA Cup heroes of going missing after crashing back down to earth with defeat.
A week after beating Birmingham in the third round, Huddersfield were soundly beaten in front of their own fans.
Ritchie said: "We simply didn't turn up. I'm very disappointed. The players know how I feel."
Chris Brandon put the Terriers ahead but two goals from Irish striker Gary Mulligan - one from the penalty spot - and another by ex-Huddersfield hot-shot Delroy Facey sealed an impressive victory for Gills, who had Adam Miller sent off in the last minute.