That is true and it's common for the starter to go to the showers with men on. I don't think that's the biggest factor though.
Even comparing only 1st time facing hitters isn't apples to apples because of pacing vs. all-out approaches of roles. I do believe 1st time through is an entire flip-turn of who looks better. What we really should be talking about is 3rd+time facing for starters vs 1st for middle/long relievers since those are the options. I guess you could lump in late inning guys and closers if you are thinking of advocating to bring them in in the 5th or 6th? Or you have a bullpen like early 2000s Ms, late 90's Yankees or late 80s A's or Reds. For most teams ever the choices in the 6th or so have largely been your worst relievers vs the guy that started.