I presume it took some effort to move the stock and that the barrel is free floating. I'd also leave it alone, but keep an eye on it during weather changes to make sure the wood doesn't warp and touch the barrel. Might also check interference if you use a rifle sling to brace your aim. But, if you are competent with woodworking tools and finishing and don't like the look, then go at it to even it up, but not all the way to the action.

Don't even think about messing with that stock! It's getting groups like that because the barrel is "free-floated" and not under stress. If you try to "fix" it, you'll just make it real bad.

If it shoots, don't touch it. If the forearm pressing against the stock ruins the accuracy, you will have to get it back to where it was or rasp out the forend to free float the barrel. That may cause you to have to glass bed the action if it isn't already solid. Good luck.