yes this could have upset him and caused him to go and hide until they have gone. Cats can be very senitive about any sort of changes to their world. He may have not socialised with a lot of people before or could just be that he doesn't like the scent they wear. I have come across this before.
I can't see why he wouldn't come back once he thinks the intruders have gone. I wouldn't worry too much, just make sure that you leave some food out for him so that he still feels reassured that this is his home.
