Judges 20:36

Then the men of Benjamin saw that they had lost the battle.

The men of Israel had moved away from their positions in front of Benjamin. They had depended on the men they had hidden near Gibeah.