Irritation mixed with feeling troubled, bothered, or put-upon..
The state of being vexed — irritation blended with being troubled, worried, or harassed; also a cause of such trouble.
Overlaps heavily with annoyance and irritation but is distinctly more literary, formal, and old-fashioned in register (at home in elevated or biblical prose — “vanity and vexation of spirit”). Connotatively it adds a sense of being beset or burdened over time, more than the momentary sting of irritation, owing to its root vexare “to harass, shake.” Unlike pique it has no wounded-pride trigger; unlike frustration it lacks the goal-blockage element; gentler and more archaic than exasperation, indignation, or the high-band anger words.
c. 1400, “legal harassment”; early 15c. “mental distress,” from Latin vexationem “annoyance, harassing; distress,” from vexare “to harass, trouble.” By mid-15c. also “a cause of irritation.”
The legal-harassment sense (c. 1400) predates the broader “mental distress / irritation” sense; both the harassing and the “cause of trouble” senses survive. The main modern movement is toward more formal/literary use — an observation from usage rather than an explicitly sourced register shift. No recent-generation semantic shift is sourced.