This behavior of motor mostly due to sudden change in current i.e. for a very small change in motor supply voltage through DAC (i.e. 15000 to 16000), there is huge change in current supplied to motor which is not the case in PWM as seen from your experiments. This may be due to non-linearity in DAC output or amplifier after it. If you are using transistor as output stage check whether it is operating in its linear region, if not change it. Remember that, in case of PWM transistor acts as switch which will provide average voltage and current and its linearity does not matter but when DAC output is given, transistor operates in its active region rather than switch, this is where it can be non-linear. SO, i suggest replace transistor with high gain and high output current op-amp if motor is small or else change the transistor which operates in entire region of DAC output.
If possible can you make a block diagram at least (schematic will be good) and share it. It will be more clear and solution will be obvious, if design is here.
Also, motor info is required too, plz consult your professor and find out if its 3-phase permanent magnet BLDC motor, or self start dc motor etc.
Hope that helps.