If it's not RR tight at the beginning of the run, then gets worse throughout the run... it's probably mostly due to TIRES. You can try more left, less caster, etc, but if the tires aren't right it will still give you fits.
What kind of kart is it? What are your full set up numbers?
Need kart type, year and tire type to help. Also mention track condition, is it wet? Dry and hard? etc. All details will help those in the know steer you in the right direction!
Pushed bad where, on entry, middle or off ?
Where is your cross set, and RF camber,
You picked up 4th because it would turn however now the timing of the transfers are off not working in harmony.
As James points out biggest reason for the push initially is most likely tires, with a smaller chassis tweak needed.