In achieving your aim (of pouring away the bath water), make sure you don’t do
something else (pouring away the baby) that more than cancels out your
achievement.

> A certain Mr. Williamson, who had lived for years in London, decided to
break with the past and seek a new life overseas. The day before he was due to
sail he destroyed a pile of old correspondence, only to discover when it was
to late that he had burnt his passport as well.

We can say of the careless Mr. Williamson that he had emptied the baby out
with the bath water.

Another meaning of the prover is to reject all of something, including the
good, because part of it is bad.