Rewrite DB refresh script
See !11. The DB refresh script on the production pipeline keeps failing with either:
- Error:
2023-05-26T01:43:58 : ERROR: pika.adapters.base_connection - connection_lost: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",)
2023-05-26T01:43:58 : ERROR: pika.adapters.blocking_connection - Unexpected connection close detected: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",)
Traceback (most recent call last):
File "/work/refresh_missing_db_records.py", line 137, in <module>
sys.exit(main())
File "/work/refresh_missing_db_records.py", line 52, in main
publish_amqp_messages(added_file_gen)
File "/work/refresh_missing_db_records.py", line 128, in publish_amqp_messages
channel.basic_publish(
File "/usr/local/lib/python3.8/dist-packages/pika/adapters/blocking_connection.py", line 2265, in basic_publish
self._flush_output()
File "/usr/local/lib/python3.8/dist-packages/pika/adapters/blocking_connection.py", line 1353, in _flush_output
self._connection._flush_output(lambda: self.is_closed, *waiters)
File "/usr/local/lib/python3.8/dist-packages/pika/adapters/blocking_connection.py", line 523, in _flush_output
raise self._closed_result.value.error
pika.exceptions.StreamLostError: Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')
- OOM: I think s3fs is holding on to too much (
🤷 ) and causing the script to get killed.
My guess is this is because of two things:
- The script is not asyncronous so pika gets conflicted imports from the aio version of the library that is also imported in these scripts.
- The S3 bucket has too many files in it so s3fs is globbing for a large amount of files and python can't handle them all. If there was an easy way to paginate then maybe it'd be better.