(third-person singular simple present screws the pooch, present participle screwing the pooch, simple past and past participle screwed the pooch)
1950s, from earlier fuck the dog (“fritter, waste time”) (1935) (compare fuck around), later sense of “make an embarrassing mistake” (compare screw up, fuck up). Popularized by use by Tom Wolfe in The Right Stuff (1979), and film adaptation The Right Stuff (1983).
The term was first documented in the early "Mercury" days of the US space program. It came there from a Yale graduate named John Rawlings who helped design the astronauts' space suits. The phrase is actually a bastardization of an earlier, more vulgar and direct term which was slang for doing something very much the wrong way, as in "you are fucking the dog!" At Yale a friend of Rawlings', the radio DJ Jack May (a.k.a. "Candied Yam Jackson") amended this term to "screwing the pooch" which was simultaneously less vulgar and more pleasing to the ear.
The term, however, did not enter the popular lexicon until Tom Wolfe used it in his book about the space program, The Right Stuff, where it was used to describe a supposed mistake by astronaut Gus Grissom.