Filtering instruments from songs is more of a voodoo art than a hard science. It works perfectly with some songs, fairly well with other songs, and still there are songs where it doesn't work at all. In short, results vary widely for different songs.
The basic technique revolves around subtracting what is common to both the left and right channels. As such, this works extremely well to remove vocals from a song because they are almost always panned in the dead center. It also quite often works very well to remove bass for the same reason. However, when songs are recorded and mixed, it's ultimately up to the audio engineers working on the song to determine where things are placed in the mix, and what effects are on the different tracks.
Vocal removal from songs is generally very good. Please refer to Removing Vocals from Music for more information.
The purpose here is to outline some basics about how to filter instruments or remove instruments from songs. It is assumed here that the instrument is center panned, but this is often not true. See Center vs. Non-center Panned Audio Removal and Shifting the Sound Stage for information to help in those cases.
The first thing to understand is that the actual sound of an instrument is much more complex than a simple sine wave at a specific frequency that produces a pure note. This is a sine wave with a dotted line to show the root mean square (RMS):
Where a sine wave produces a pure tone at it's frequency, musical instruments don't. Musical instruments all have different sounds and characters, and a large part of this is exactly because they don't produce pure tones. They have a much more complex structure that is perhaps most easily understood by examining harmonics (multiples of a fundamental frequency) and overtones.
For example, an average 22 fret electric guitar in standard EADGBE tuning starts at E2 and extends up to D6, covering 82Hz to 1,175Hz. However, harmonics and overtones from a guitar extend higher up to around 5,000Hz.
The character of the sound is spread across a large frequency range, with higher peaks at some frequencies, and lower levels at other frequencies. Combined, all of this together is the total "energy" of the sound. From a different perspective, the total energy of a sound is spread across a range of frequencies. Taking from the guitar example above, the energy of the sound is almost entirely in the 80Hz ~ 5,000Hz range.
There is an excellent examination of musical instrument spectra above 20,000Hz by James Boyk of Caltech here that explains more about the energy of various instruments. In particular, the article contains a good number of diagrams illustrating peaks at different harmonics.
The key point to take away from the article for the purpose of filtering instruments from songs, is that the sound from instruments exists across a wide range of frequencies and not merely just at the fundamental frequency (the note played).
Also of importance is that there is a cut-off point at which the energy levels of a sound are no longer significant.
Putting that into a concrete example, when middle C is played on a guitar, the fundamental frequency is 262Hz, but there is also more energy across a larger range of frequencies, and that energy is strongest at multiples of the fundamental frequency, i.e. The energy is strongest at harmonic intervals. So if we were to eliminate 100% of all the sound at 262Hz, we would still hear the note being played, though at a lesser volume level and with a changed character.
What that means for removing instruments from songs is that when you target the frequency range of the notes being played (the fundamental frequencies ), you need to extend the frequency range higher to cover the additional energy of the sound, e.g. The harmonics.
As a rule of thumb, to properly target higher frequencies where the sound of the instrument exists, since the energy (peak level) tends to decline as the frequency increases, less and less attenuation can be applied.
Different instruments produce sounds that extend to higher frequencies by different amounts. This is even true for the same type of instrument, but made differently. i.e. A $5,000 acoustic guitar sounds different than a $79.95 guitar.
To remove instruments from a song then, several basic tools can be used. The first step is to use either center panned audio removal or non-center panned audio removal to give a basic starting point. Once the center is removed, instruments can be added back in by adjusting the equalization of 1 channel or by using various bandpass filters on 1 channel. When choosing the frequency ranges of the bandpass or equalizer then, attention must be paid to not only the notes being played, but also the amount of additional energy of the sound above the notes.
For example, say we have a song with a center panned instrument playing notes around A3 to A4 (220Hz to 440Hz). From the above, we know that there is additional sound energy above this range that we also need to target. The amount of additional sound energy is different for different kinds of instruments, and audio FX applied to them again changes things. However, once we know this, we are better prepared to adjust the sound equalizer or bandpass filter settings. So if we adjust the EQ bands between 220Hz and 880Hz, we effectively add back in the sound energy of the instrument in that range. However, this also adds back in any other instruments or vocals in that range.
So instead of adjusting the EQ bands between 220Hz and 880Hz, we may get better results by adjusting the 440Hz to 880Hz range, or 440Hz to 1,760Hz, or 880Hz to 1,760Hz, or whatever else works best. Experimentation with different settings is the only certain way to effectively get the best results.
However, some basic knowledge of the frequency ranges of various instruments, and the sound energy above their fundamental frequency range, can help you to more quickly and easily filter out and remove instruments from songs. The Independent Recording Network (IRN) has an absolutely excellent "Interactive Frequency Chart" that tells you the fundamental frequency ranges for many instruments, the higher frequency ranges for their overtones and harmonics, the sensitivity of the human ear to different frequency ranges, characteristics of the different instruments, and a few other good things to know. Click the thumb of the Interactive Frequency Chart below to zoom:
There is a link in the IRN Interactive Frequency Chart to another excellent chart on the sensitivity of human hearing. Having at least a basic understanding about how your ear works can help you make the right adjustments when filtering instruments from songs.