I'm in a similar situation to Greg, in that my biggest/best clients act as 'middle men' between me and the end users/agencies etc. (I do a lot of corporate work, and the middle men are the event production companies). They deal with all the stuff that I don't like - meeting with clients, filtering through assets removing all the crap and just give me a nice concise brief with the assets that I need, and I'm left to get on with what I do best.
They also manage the clients' expectations, in terms of what's possible in a given timeframe/budget, when things will be delivered etc. They kind of act like an interface between me and 'corporate people', translating their buzzword-filled language into something I can work with, and turning my, often blunt, comments into something less 'abrasive' to present to the client!
They charge their clients almost double what I charge them for motion graphics, but that's fine by me because they're handling all the crap that I don't want, and I'm still earning an hourly rate that I'm happy with.
Like Greg, I sometimes turn regular clients down because I have too much other work on (or at least, I tell them it will be x days/weeks before I can start on it) and they go elsewhere if that won't work for them. They still come back to me with subsequent jobs, presumably because I provide a good service at a reasonable price and they like working with me. They also know that I sometimes get heavily booked up with work, and that they need to give me plenty of notice if they want to guarantee my availability for a job - so they don't promise things to their clients until they've checked my availability.
I used to work from home, starting early in the morning and working until late at night, but now I work from a separate office a good distance form home and stick (most of the time) to an ~8 hour day. It hasn't done my income any harm, as I seem to work more productively from my office than I did from home, and it's done my health the world of good - I suffered badly from stress in the past, but I've got on top of that by not allowing myself to be overstretched.
Thankfully downtime is very rare for me these days, but when I get it, I use it to catch up with accounts, tidy up hard drives, find ways to improve workflow, do a bit of experimental work/learning (a rare treat these days and something I wish I could do more often), and develop new 'off-the-shelf' products to add to my range. I've actually promised myself this year that I'm going to spend one day a month doing purely experimental stuff / learning so I don't get stuck in a rut in terms of style/ability,
Phil