Everton dispatched Thomas Frank’s men 4-1 to advance to the fifth round, with goals from Yerry Mina, Richarlison, Mason Holgate and Andros Townsend.
Lampard was appointed as the new Toffees boss on Monday, replacing Rafael Benitez in the role.
Lampard’s appointment has brought an air of optimism around Goodison Park which was evident with the performance of the players and reception from the fans.
"It's been brilliant this week, very special," Lampard told a news conference.
"It's a day that I'll never forget. It's only the start in terms of my time here but I can't help but get caught up in the emotion of the moment. The way the players approached the game meant that the fans bought into it straight away. It was a very special day for me.
"I want to be the best I can be, and if the fans can see it all well and good. They want to see their manager giving everything and that's just how I work.
"I want to be there to support the players when they make passes or play through lines, or make tackles, covering distance.
"Those things are special and what we work towards. As a manager today I was proud of the performance and it did get me excited. Now I need to calm down and work towards the league games but we saw hopefully some nice signs that we can take forward."
Lampard’s first Premier League game will be against Newcastle United on Tuesday.
