Your crawler's volume is high enough that you should switch to a more efficient consumption method. See https://developer.wordpress.com/docs/firehose/ for more information.