Finite state machines is covered by discrete mathematics and directly related to Boolean algebra, Set theory, Graph Theory and indirectly Coding Theory.
This book may be useful:
Handbook of finite state techniques (Discrete mathematics & its applications)
Bruce W. WATSON