State your source
Please provide your source where passports will be required to enter federal facilities and fly from state to state. Common Access Cards (CACs) are given to all gov’t employees and contractors with regular business there once they have been cleared. Are you referring to those who don’t have regular business in the federal facility?
If what you say is true, then, for example, you can’t enter a post office (PO), a federal facility, to conduct business which would be illegal since it is against the commerce clause of the Constitution. Just because you have UPS and FedEx to ship packages, mailing letters, cards, etc is only done through the PO and cannot be prohibited (which I will caveat with I am not sure about postal crimes, but that is not the point here).
Since when is traveling from state to state via an airplane going to require a passport? You already have the federal ID law that you must have a certain type of ID as required by DHS requirements (and some states have received waivers on).
Again, please state your source, chapter and verse specifically if you don’t mind.