Hi
The gerber and drill file have file formats which contains informations like precision, metric/imperial, gerber type, etc... So the gerber viewer automatically interprets them and views. However if you select manual format type while loading the scaling may be 10x larger or smaller and sometime the viewer can also interpret the format in incorrect scale. To check you can try in other viewer as well like viewmate.
About drill offset, the CAD software may have two different origins one for copper layer and other one for drill layer if both are not same you may see offset. Or it is also possible that while generating nc drill, some offset is entered.
Hope this helps. If you need more details or if you want me to look at it and fix you issue please let me know.