I have been married to a Polish citizen for 3 years, but so far we have lived outside of Poland. Can I apply for permanent residence in Poland?

Unfortunately no. You will be able to apply for permanent residence only after 2 years of uninterrupted residence in Poland on the basis of a temporary residence permit granted in connection with being married to a Polish citizen. If you have never applied for a temporary residence in Poland before and you intend to live permanently in Poland, you must first apply for a temporary residence.