'In the Wilderness' in the Bible

harden not your hearts, as in the provocation, in the day of temptation in the wilderness, when your fathers tempted me,

But with whom was He provoked forty years? was it not with those that sinned, whose carcases fell in the wilderness?