Not much stresses me about brewing; I prep my strike and sparge water the day before, and mill the grains on brewday while the strike water heats. Hops get weighed out while the mash rests.
In my specific case, my only real stress is getting my wife and son out of the house on time (I brew during the workday, and since I need the garage to brew, I can't get started until they leave in the morning).
One other stress is staying on schedule, getting the brewday done, and everything cleaned up before my wife gets home.
This is particularly important on those brewdays when it's raining or snowing out, as I've found out (the hard way) that she's not amused about parking in the driveway on a rainy/snowy day b/c my brewday went into overtime.