As the voltage gets lower the noise will tend to get higher in frequency. The regulator continues to work even after power has been removed by sucking all of the juice from the surrounding capacitors. In almost all cases this is caused by a faulty voltage regulator that isn't shutting down as its supposed to.
What is happening is one of the components is vibrating to the point of making sound, normally an inductor. The capacitors are powering it long enough to cause the noise. In fact there in almost everything electronic now. These are located throughout a computer in the PS, MB, HDD, CDROM, Graphics cards and more.
What your hearing most likely is a faulty switching voltage regulator.
