René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Deadlock in Defects-issue783-offload1 test","articleBody":"The scan is stuck because it can't find the range:\n\n\u003cpre\u003e\nRANGE SERVER range not found - (a) 2[0098861..??]\n\u003c/pre\u003e\n\nThis range is in the process of being loaded by rs3, but it can't update the METADATA table.  Here's the stack trace:\n\n\u003cpre\u003e\nThread 77 (Thread 0x7f726c809700 (LWP 1474)):\n#0  0x000000399d4df1b3 in poll () from /lib64/libc.so.6\n#1  0x00000000007a5485 in Hypertable::TableMutatorAsyncScatterBuffer::create_redo_buffer(unsigned int) ()\n#2  0x000000000079ab3f in Hypertable::TableMutatorAsync::buffer_finish(unsigned int, int, bool) ()\n#3  0x00000000007a2bbd in Hypertable::TableMutatorAsyncScatterBuffer::finish() ()\n#4  0x00000000008337b5 in Hypertable::TableMutatorAsyncHandler::run() ()\n#5  0x0000000000794b36 in Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsync*) ()\n#6  0x0000000000795843 in Hypertable::TableMutator::flush() ()\n#7  0x0000000000592565 in Hypertable::Apps::RangeServer::load_range(Hypertable::ResponseCallback*, Hypertable::TableIdentifier const\u0026, Hypertable::RangeSpec const\u0026, Hypertable::RangeState const\u0026, bool) ()\n#8  0x00000000006a9d9b in Hypertable::RangeServer::Request::Handler::LoadRange::run() ()\n#9  0x0000000000566c35 in Hypertable::ApplicationQueue::Worker::operator()() ()\n#10 0x00007f72819dc082 in thread_proxy () from /opt/hypertable/doug/0.9.8.5/lib/libboost_thread.so.1.54.0\n#11 0x000000399d8079d1 in start_thread () from /lib64/libpthread.so.0\n#12 0x000000399d4e89dd in clone () from /lib64/libc.so.6\n\u003c/pre\u003e\n\n\nOver in rs1, it appears that the METADATA range `0/0[0/0:??..??]` is stuck at the end of run_compaction blocked on the LiveFileTracker mutex.  Here's a snippet of the rangeserver dump:\n\n\u003cpre\u003e\nRANGE 0/0[0/0:??..??]\n...\nstate=3\n\u003c/pre\u003e\n\nState 3 is RELINQUISH_LOG_INSTALLED and here's the stack trace from rs1:\n\n\u003cpre\u003e\nThread 34 (Thread 0x7f1964493700 (LWP 1575)):\n#0  0x000000399d4df1b3 in poll () from /lib64/libc.so.6\n#1  0x00000000007a5485 in Hypertable::TableMutatorAsyncScatterBuffer::create_redo_buffer(unsigned int) ()\n#2  0x000000000079ab3f in Hypertable::TableMutatorAsync::buffer_finish(unsigned int, int, bool) ()\n#3  0x00000000007a2bbd in Hypertable::TableMutatorAsyncScatterBuffer::finish() ()\n#4  0x00000000008337b5 in Hypertable::TableMutatorAsyncHandler::run() ()\n#5  0x0000000000794b36 in Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsync*) ()\n#6  0x0000000000795843 in Hypertable::TableMutator::flush() ()\n#7  0x000000000067cbdc in Hypertable::MetadataNormal::write_files(std::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e const\u0026, std::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e const\u0026, long, unsigned int) ()\n#8  0x0000000000656ecd in Hypertable::LiveFileTracker::update_files_column() ()\n#9  0x00000000005dcf60 in Hypertable::AccessGroup::run_compaction(int, Hypertable::AccessGroup::Hints*) ()\n#10 0x00000000006964c5 in Hypertable::Range::relinquish_compact() ()\n#11 0x0000000000696e18 in Hypertable::Range::relinquish() ()\n#12 0x00000000005b01df in Hypertable::MaintenanceQueue::Worker::operator()() ()\n#13 0x00007f19950ac082 in thread_proxy () from /opt/hypertable/doug/0.9.8.5/lib/libboost_thread.so.1.54.0\n#14 0x000000399d8079d1 in start_thread () from /lib64/libpthread.so.0\n#15 0x000000399d4e89dd in clone () from /lib64/libc.so.6\n\u003c/pre\u003e\n\nI've also seen it here:\n\n\u003cpre\u003e\nThread 34 (Thread 0x7f1964493700 (LWP 1575)):\n#0  0x000000399d80e264 in __lll_lock_wait () from /lib64/libpthread.so.0\n#1  0x000000399d809508 in _L_lock_854 () from /lib64/libpthread.so.0\n#2  0x000000399d8093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0\n#3  0x0000000000566028 in boost::mutex::lock() ()\n#4  0x0000000000656cc4 in Hypertable::LiveFileTracker::update_files_column() ()\n#5  0x00000000005dcf60 in Hypertable::AccessGroup::run_compaction(int, Hypertable::AccessGroup::Hints*) ()\n#6  0x00000000006964c5 in Hypertable::Range::relinquish_compact() ()\n#7  0x0000000000696e18 in Hypertable::Range::relinquish() ()\n#8  0x00000000005b01df in Hypertable::MaintenanceQueue::Worker::operator()() ()\n#9  0x00007f19950ac082 in thread_proxy () from /opt/hypertable/doug/0.9.8.5/lib/libboost_thread.so.1.54.0\n#10 0x000000399d8079d1 in start_thread () from /lib64/libpthread.so.0\n#11 0x000000399d4e89dd in clone () from /lib64/libc.so.6\n\u003c/pre\u003e\n\nI also see a number of threads (in rs1) stuck here:\n\n\u003cpre\u003e\nThread 87 (Thread 0x7f19862e3700 (LWP 1418)):\n#0  0x000000399d80e264 in __lll_lock_wait () from /lib64/libpthread.so.0\n#1  0x000000399d809508 in _L_lock_854 () from /lib64/libpthread.so.0\n#2  0x000000399d8093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0\n#3  0x00000000005660b8 in boost::unique_lock\u003cboost::mutex\u003e::lock() ()\n#4  0x000000000065669f in Hypertable::LiveFileTracker::remove_references(std::vector\u003cstd::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e, \nstd::allocator\u003cstd::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e \u003e \u003e const\u0026) ()\n#5  0x00000000005d6c40 in Hypertable::AccessGroup::release_files(std::vector\u003cstd::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e, std::all\nocator\u003cstd::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e \u003e \u003e const\u0026) ()\n#6  0x000000000066c057 in Hypertable::MergeScannerAccessGroup::~MergeScannerAccessGroup() ()\n#7  0x000000000066c3b9 in Hypertable::MergeScannerAccessGroup::~MergeScannerAccessGroup() ()\n#8  0x0000000000671556 in Hypertable::MergeScannerRange::~MergeScannerRange() ()\n#9  0x0000000000563325 in std::_Sp_counted_base\u003c(__gnu_cxx::_Lock_policy)2\u003e::_M_release() ()\n#10 0x000000000058f898 in Hypertable::Apps::RangeServer::create_scanner(Hypertable::RangeServer::Response::Callback::CreateScanner*, Hypertable::TableIdentif\nier const\u0026, Hypertable::RangeSpec const\u0026, Hypertable::Lib::ScanSpec const\u0026, Hypertable::QueryCache::Key*) ()\n#11 0x00000000006a0bc5 in Hypertable::RangeServer::Request::Handler::CreateScanner::run() ()\n#12 0x0000000000566c35 in Hypertable::ApplicationQueue::Worker::operator()() ()\n#13 0x00007f19950ac082 in thread_proxy () from /opt/hypertable/doug/0.9.8.5/lib/libboost_thread.so.1.54.0\n#14 0x000000399d8079d1 in start_thread () from /lib64/libpthread.so.0\n#15 0x000000399d4e89dd in clone () from /lib64/libc.so.6\n\u003c/pre\u003e\n\nand here ...\n\n\u003cpre\u003e\nThread 70 (Thread 0x7f197b8d2700 (LWP 1456)):\n#0  0x000000399d80e264 in __lll_lock_wait () from /lib64/libpthread.so.0\n#1  0x000000399d809508 in _L_lock_854 () from /lib64/libpthread.so.0\n#2  0x000000399d8093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0\n#3  0x00000000005660b8 in boost::unique_lock\u003cboost::mutex\u003e::lock() ()\n#4  0x0000000000656a36 in Hypertable::LiveFileTracker::add_references(std::vector\u003cstd::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e, std\n::allocator\u003cstd::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e \u003e \u003e const\u0026) ()\n#5  0x00000000005db064 in Hypertable::AccessGroup::create_scanner(boost::intrusive_ptr\u003cHypertable::ScanContext\u003e\u0026) ()\n#6  0x000000000068b8ef in Hypertable::Range::create_scanner(boost::intrusive_ptr\u003cHypertable::ScanContext\u003e\u0026, std::shared_ptr\u003cHypertable::MergeScannerRange\u003e\u0026) \n()\n#7  0x000000000058f713 in Hypertable::Apps::RangeServer::create_scanner(Hypertable::RangeServer::Response::Callback::CreateScanner*, Hypertable::TableIdentif\nier const\u0026, Hypertable::RangeSpec const\u0026, Hypertable::Lib::ScanSpec const\u0026, Hypertable::QueryCache::Key*) ()\n#8  0x00000000006a0bc5 in Hypertable::RangeServer::Request::Handler::CreateScanner::run() ()\n#9  0x0000000000566c35 in Hypertable::ApplicationQueue::Worker::operator()() ()\n#10 0x00007f19950ac082 in thread_proxy () from /opt/hypertable/doug/0.9.8.5/lib/libboost_thread.so.1.54.0\n#11 0x000000399d8079d1 in start_thread () from /lib64/libpthread.so.0\n#12 0x000000399d4e89dd in clone () from /lib64/libc.so.6\n\u003c/pre\u003e\n","author":{"url":"https://github.com/nuggetwheat","@type":"Person","name":"nuggetwheat"},"datePublished":"2015-02-25T00:54:34.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":15},"url":"https://github.com/31/hypertable/issues/31"}