The modern sense of screwed originates in the mid-1600s with a sense of to screw as a means of "exerting pressure or coercion", probably in reference to instruments of torture (e.g. thumbscrews). It quickly gained a wider general sense of "in a bind; in unfortunate inescapable circumstances".
The above is what Wiki says. Seems plausible.
I always thought that a screw is something more permanent than a nail and that is difficult to extract.
As an aside, screwing is used in english in reference to sex, yet other european languages use 'nailing' in the same context.![]()