For some reason, D&D has gotten it into its head that men-at-arms would be 0-level. This trope dates back pretty far, and is taken as canon by many.
Let's examine this for a minute, though.
The historical term "man-at-arms" was used to denote a professional cavalryman in full armour. Knights would be an example of a man-at-arms, but many men-at-arms were not knights. In D&D, however, the term tends to be used to refer to any professional soldier, regardless of equipment.
In either case, though, the term refers to a professional.
The idea that a professional solder has the same odds of hitting in combat as an untrained peasant is, frankly, laughable. This professional, in fact, has lower odds of hitting in combat than your fresh-faced 1st-Level Fighter with 0 XP.
In an older edition like BECMI (my old stand-by), a professional soldier with training and experience should probably be around level 3 - greater odds of hitting, and much more survivable than an untrained peasant.
On average, a peasant (AC 9, 1-6 HP, Attack +0) will take around 5 rounds to kill a Level 3 fighter (AC 9, 3-18 HP, Attack +3). The Level 3 fighter will only take around 1 or 2 rounds to kill the peasant. The difference swells, obviously, if you give the man-at-arms armour.
Conclusions? City watchmen, mook guards, anyone with no real training - 0-level or Fighter 1.
Serious men-at-arms, with training and experience? Probably Fighter 3-5.