Invalid HFile block magic

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Invalid HFile block magic

Ted Tuttle-2
Hello-

We are running HBase 1.2.0 from cdh 5.7.0 distribution.

For about 10 of our 200 tables we are seeing client errors like this: https://pastebin.com/wDdyd1cZ

This stack trace led me to this JIRA: https://issues.apache.org/jira/browse/HBASE-11625

It seems this is a bug that has been fixed but we don't have the patch. Unfortunately, immediate patching of our cluster is not possible.

The 2 comments in the JIRA mention major compaction of involved regions as a workaround.  Is this a legitimate workaround?  While I've had some point success here, compacting a large table w/ this problem was not effective.

What else can we do to clean up the cluster?  hbck identifies no issues w/ tables that have this error.

Thanks,
Ted

PS We moved our datacenter some weeks back.  Post-move the cluster was not immediately stable.  At least once we lost enough nodes that there weren't enough nodes to host all our regions and the cluster when down completely. After settling in, we experienced 2 weeks of stability before aforementioned problems began.