Money for the most part for a lot of people.
Passports are $400+ USD, then there are the plane tickets, which are hundreds of dollars. Then to top it off you need to have room and board while looking for a job and someplace to live.
Another thing I’ve heard is fear of leaving the known and family.
Pretty much the only time we need passports is if we travel outside the U.S. and territories. Those that take cruises or cross borders to other countries would, but generally speaking a majority of Americans don’t have passports.