"Middle Men" was a term used by the Númenóreans for Men of Middle-earth who were related to the Edain, the ancestors of the Númenóreans themselves. The proper term is "Men of Twilight" (as opposed to the Men of Gondor, "Men of Light", and the Haradrim and Easterlings, "Men of Darkness" etc.).
When the Númenóreans returned to the coasts of Middle-earth in the Second Age, they found a people who spoke languages which were distantly related to the Númenórean tongue, Adûnaic. Númenórean scholars declared that this was because these Men were descendants of the fathers of the Edain, the Atanatári, who had not crossed the Ered Luin and entered Beleriand during the First Age. The Númenóreans set up friendly relations with them, and declared them to be Middle Men, as opposed to the High Men (the Edain) or the Men of the Shadow, meaning those hostile to Númenor or in the service of Sauron.
The Northmen of Rhovanion were counted as Middle Men, as were most people living in Eriador. During the Third Age the term Middle Men was still applied to the Men of Bree, and the remaining Northmen, such as the Men of Dale and Esgaroth, the Lake-town, and of course the Rohirrim. King Valacar of Gondor became so friendly with the Middle Men of Rhovanion that he married Vidumavi; a princess of the Northmen, and his son Eldacar was of mixed blood. This led to the disastrous Kin-strife of Gondor.
Most Middle Men had kinship with the House of Bëor or Marach. There were also Men in Middle-earth related to the Second House, the Haladin — Men of Enedwaith and Minhiriath, later to be known as the Dunlendings. But because they spoke a language that was unknown to the Númenóreans they were not at first recognised as Middle Men. Hostility developed between these Middle Men and the Númenóreans, which would endure until the end of the Third Age.