The color of the portal follows your Primary Color but not influence what workers that can be remotely placed.