I know I’m digging up an old bone, but this topic has been an interest of mine, and I feel that I should add my two cents.
Historically, -or refered specifically to human agents, while -er could be both human or not. Thus operator and vendor are more appropriate when talking about humans.
These days, -er is used almost exclusively to form new words, and -or is dying out. Probably, the (not simple, but easiest) rule to follow is: if you can’t find anything that attests the word should end in -or (i.e. a dictionary entry ending in -or), use -er.
[If you’re into etymology -or and -er are both descended from French: -or from -eur and -er from -ier. The distinction still stands somewhat in French]