Migrate from psycopg2 to psycopg3

psycopg3 is mostly built as a drop-in upgrade from psycopg2.
Differences are detailed here: https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html

A few highlights:

* Cancelling running queries

One major DX improvement is that Ctrl-C can now cancel a running
query. With psycopg2, the python process can be killed but a running
query usually could not be killed (without knowing the PID and running
`pg_cancel_backend(pid)` from a separate connection), to the immediate
regret of all we who sometimes accidentally execute an unfortunately
large query.

See https://www.psycopg.org/psycopg3/docs/advanced/async.html#interrupting-async-operations

* Breaking changes to parameterization

A few parameterization patterns that worked with psycopg2 no longer
work with psycopg3. They are detailed on the migration guide linked
above, but these in particular should be highlighted:

** You can no longer use `IN %s` with a tuple

This would work with psycopg2 but will not work with psycopg3:
`conn.execute("SELECT * FROM foo WHERE id IN %s", parameters=[(10,20,30)])`

Instead, this format is advised:
`conn.execute("SELECT * FROM foo WHERE id = ANY(%s)", parameters=[[10,20,30]])`

** You can no longer use `IS %s`
This would work with psycopg2 but will not work with psycopg3:
`conn.execute("SELECT * FROM foo WHERE field IS %s", [None])`

Instead, `IS NOT DISTINCT FROM %s` should be used
`conn.execute("SELECT * FROM foo WHERE field IS NOT DISTINCT FROM %s", [None])`

* Breaking changes to psycopg2 `copy` methods

`copy` methods are refactored and consolidated in psycopg3. See the
migration guide for details.

Author: austinweisgrau

parsons/databases/redshift/redshift.py

@@ -8,8 +8,7 @@
 from parsons.databases.alchemy import Alchemy
 from parsons.utilities import files, sql_helpers
 from parsons.databases.database_connector import DatabaseConnector
-import psycopg2
-import psycopg2.extras
+import psycopg
 import os
 import logging
 import json
@@ -128,11 +127,14 @@ def connection(self):
         ``with rs.connection() as conn:``
 
         `Returns:`
-            Psycopg2 ``connection`` object
+            Psycopg ``connection`` object
         """
+        # Redshift unicode encoding doesn't work out of the box with psycopg3
+        # See https://github.com/psycopg/psycopg/issues/122#issuecomment-1272352650
+        os.environ["PGCLIENTENCODING"] = "utf-8"
 
-        # Create a psycopg2 connection and cursor
-        conn = psycopg2.connect(
+        # Create a psycopg connection and cursor
+        conn = psycopg.connect(
             user=self.username,
             password=self.password,
             host=self.host,
@@ -149,7 +151,7 @@ def connection(self):
 
     @contextmanager
     def cursor(self, connection):
-        cur = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)
+        cur = connection.cursor(row_factory=psycopg.rows.dict_row)
         try:
             yield cur
         finally:

requirements.txt

@@ -4,7 +4,7 @@ boto3>=1.17.98
 boxsdk==2.10.0
 civis==1.16.0
 slackclient==1.3.0
-psycopg2-binary==2.9.3
+psycopg[c]==3.1.18
 xmltodict==0.11.0
 gspread==3.7.0
 oauth2client==4.1.3

setup.py

@@ -44,8 +44,8 @@ def main():
             "newmode": ["newmode"],
             "ngpvan": ["suds-py3"],
             "mobilecommons": ["bs4"],
-            "postgres": ["psycopg2-binary", "SQLAlchemy"],
-            "redshift": ["boto3", "psycopg2-binary", "SQLAlchemy"],
+            "postgres": ["psycopg[c]", "SQLAlchemy"],
+            "redshift": ["boto3", "psycopg[c]", "SQLAlchemy"],
             "s3": ["boto3"],
             "salesforce": ["simple-salesforce"],
             "sftp": ["paramiko"],