Although high-end retailers such as Saks and Neiman Marcus have fared well the last few years, the severity of the current economic downturn has affected luxury spending along with other sectors.