CNAME change and server shutdown does not reconnect to new server IP as expected
RickK conducted a server transfer, reassigning the "mq4" CNAME to a different physical machine, and then shutting down the old rabbitmq server.
This results in exceptions in the multi-server configuration's handling of that server (via a worker process), rather than the expected stand-back-30s-and-connect-to-new-server that would be expected.
mq4.ssec.wisc.edu oper@csppdelta.ssec.wisc.edu:/home/oper/make-aitf-L2-products/bin/make-aitf-L2-products.py /data/goes/grb/goes16/2023/2023_08_10_222/abi/L2/AIT/CMIPM2/CG_ABI-L2-CMIPM2-M6C01_G16_s20232221621550_e20232221622008_c20232221622244.nc
ERROR:pika.adapters.blocking_connection:Unexpected connection close detected: ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
ERROR:__main__:exception in server connection mq4.ssec.wisc.edu:
Traceback (most recent call last):
File "/home/rickk/amqpfind/amqpfind.py", line 303, in worker_main
amqp.consume(queue_callback)
File "/home/rickk/amqpfind/amqpfind.py", line 245, in consume
self.channel.start_consuming()
File "/home/rickk/lib/python_localenv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1866, in start_consuming
self._process_data_events(time_limit=None)
File "/home/rickk/lib/python_localenv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 2027, in _process_data_events
self.connection.process_data_events(time_limit=time_limit)
File "/home/rickk/lib/python_localenv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 825, in process_data_events
self._flush_output(common_terminator)
File "/home/rickk/lib/python_localenv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 522, in _flush_output
raise self._closed_result.value.error
pika.exceptions.ConnectionClosedByBroker: (320, "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'")
Process Process-1:
Traceback (most recent call last):
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/rickk/amqpfind/amqpfind.py", line 310, in worker_main
amqp.close()
File "/home/rickk/amqpfind/amqpfind.py", line 264, in close
self.channel.close()
File "/home/rickk/lib/python_localenv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1514, in close
self._impl._raise_if_not_open()
File "/home/rickk/lib/python_localenv/lib/python3.6/site-packages/pika/channel.py", line 1389, in _raise_if_not_open
raise exceptions.ChannelWrongStateError('Channel is closed.')
pika.exceptions.ChannelWrongStateError: Channel is closed.