Good question Thierry & good, cheap solution Liam.
I must say I've looked at various boat's worn gudgeons / pintles and generally thought "It's taken them 20yrs to get to this point, think I'll leave well alone!"
However, the wear, particularly if uneven could potentially very slightly affect the geometry of the rudders.
As far as I can see on my boat, the gudgeon castings are symmetrical, so turning them over seems a good idea & perhaps also implement Liam's suggestion to minimise future wear. Adding washers will be easy on the bottom pintles as gravity will keep them there. A bit of glue probably necessary for the top sets.