iPad is a bit curved

That's a tough one. On one hand it could be fine for years even with the bend, or the screen could eventually just snap one day out of the blue from the stress; there's just no way of knowing what it will do.

If it was mine personally I would take the screen off and try to straighten it; your instincts are right on the money with regard to not trying to straighten it with the screen in place. Here's the guide you'd need to do that.

iPad Air 4 Screen Replacement - iFixit Repair Guide

Take the screen off then use a straight edge - something like a metal ruler works well - and gently bend the frame until it's straight. I've done this with an iPad Mini my daughter-in-law was given; in her case the screen had already broken but otherwise it was exactly the same operation. Bend it a bit then check top, bottom, left, right, up and down with the ruler to make sure it's perfectly flat in all dimensions; then and only then you're ready to put the screen back on. Make sure you pre-fit the screen before gluing it down; it should slip into place easily without being forced - if you have to make it go in, then you may have to bend some of the edge away from the glass to give it more breathing space.

Of course, there's always a risk that just the act of trying to remove the screen could possibly break it, so keep that in mind when deciding which way you want to go. It's not a high probability, but it could happen so you should be aware of it.