Sounds very much like a loose crank nut, which is a known problem and seems to effect a random selection of XTs over the years.
What you will need to do is remove the left hand engine cover (with water and oil pumps) and expose the primary drive gears. You can then access the 36mm nut on the end of the crankshaft, fit a new lock washer and re-torque it.
This thread is probably worth a read:
http://www.xt660.com/showthread.php?t=23951
It isn't an expensive job to do (just new gaskets etc.) unless the loose nut has caused other damage. In all the cases recorded on the forum, a re-tightening of the nut has rectified the issue with no further trouble.