When you build a circuit without understanding it, it's not surprising it doesn't work properly.;-)
If you look at the data sheet for the ADC0804LCN you will see that the input signal goes from ground to V+, thus you must bias a normal audio signal, which goes above and below ground, at 1/2 V+ to keep within this signal range.
For maximum dynamic range, the peak-to-peak audio signal for the loudest signal should go from 0V to V+.
The data sheet also shows that the free run conversion rate is 8888 conversions/sec. The Nyquist theorem states that the maximum audio signal you can convert is 1/2 that or about 4.4kHz (which is low, comparable to phone bandwidths). Any audio frequencies above that will alias into the passband causing noise and distortion. So you need a high order 4kHz low-pass active-filter (probably 4-pole or greater) at the ADC input to prevent this aliasing.
Also 8-bits is a low-number of bits for sending audio (16-bits is typical) and will give a low S/N ratio as well as some possible audible distortion.