Early 19th century US; attested 1834. In original use, often used to refer to pulling oneself over a fence, and implying that someone is attempting or has claimed some ludicrously far-fetched or impossible task. Presumably a variant on a traditional tall tale, as elaborated below. The shift in sense to a possible task appears to have developed in the early 20th century, and the use of the phrase to mean “a ludicrous task” continued into the 1920s.
Widely attributed to The Surprising Adventures of Baron Munchausen, (1781) by Rudolf Erich Raspe, where the eponymous Baron pulls himself out of a swamp by his hair (specifically, his pigtail), though not by his bootstraps; misattribution dates to US, 1901. The Adventures is primarily a collection of traditional (centuries-old) tall tales; using bootstraps presumably arose as a variant on the same tall tale, or arose independently.