A deflected strike by Pedro Obiang in the 72nd minute and a goal from Andre Ayew five minutes later, when he stabbed home a loose ball from close range after a corner, were enough to end Huddersfield's unbeaten start to the season.
West Ham were playing their first home game of the campaign after having to travel for their opening three matches because the stadium they moved into last season had been used for the world athletics championships in August.
"I've felt the pressure for a whole year now. It is maybe not logical for me to be under pressure after three games but it is modern football," said Bilic who was 49 on Monday.
"I am happy for the lads and for the club. We started with three away games and three defeats and this was a must-win game.
"We had a gameplan and from the first moment we were on the front foot. We stopped them from creating anything. We got our reward, a little bit lucky for the first goal but overall we deserved it.
"It is my birthday today and this is a great, great present for me," added the delighted Croatian.
The win took West Ham off the bottom of the table and up to 18th place, still in the relegation zone, while Huddersfield stay sixth with seven points from their four games.
Enter a name and e-mail address and we will e-mail this website link too them.