René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Bug in com.sun.java.help.search.BitSet.java leads to a long trail of exceptions","articleBody":"Our organization uses JavaHelp for packaging our help documentation for our product. We've noticed that at some point when a very large file was being indexed via jhindexer, a number of IOExceptions were generated:\n\njava.io.IOException: Can't store Document\nat com.sun.java.help.search.DefaultIndexerKit.storeToken(DefaultIndexerKit.java:164)\nat com.sun.java.help.search.DefaultIndexerKit.parseIntoTokens(DefaultIndexerKit.java:117)\nat com.sun.java.help.search.HTMLIndexerKit$HTMLParserCallback.addContent(HTMLIndexerKit.java:1092)\nat com.sun.java.help.search.HTMLIndexerKit$HTMLParserCallback.addContent(HTMLIndexerKit.java:1064)\nat com.sun.java.help.search.HTMLIndexerKit$HTMLParserCallback.handleText(HTMLIndexerKit.java:411)\nat javax.swing.text.html.parser.DocumentParser.handleText(DocumentParser.java:241)\nat javax.swing.text.html.parser.Parser.handleText(Parser.java:345)\nat javax.swing.text.html.parser.Parser.endTag(Parser.java:427)\nat javax.swing.text.html.parser.Parser.parseTag(Parser.java:1804)\nat javax.swing.text.html.parser.Parser.parseContent(Parser.java:2044)\nat javax.swing.text.html.parser.Parser.parse(Parser.java:2211)\nat javax.swing.text.html.parser.DocumentParser.parse(DocumentParser.java:105)\nat javax.swing.text.html.parser.ParserDelegator.parse(ParserDelegator.java:73)\nat com.sun.java.help.search.HTMLIndexerKit.parse(HTMLIndexerKit.java:116)\nat com.sun.java.help.search.Indexer.parseFile(Indexer.java:259)\nat com.sun.java.help.search.Indexer.compile(Indexer.java:219)\nat com.sun.java.help.search.Indexer.main(Indexer.java:87)\n\nUsing the source available from subversion, we've localized the issue to a very subtle bug in com.sun.java.help.search.BitSet.concatenate():\n\n162\\. int tp = _free - 1; // target\n163\\. int sp = 0; // source\n164\\. do\n\n{ 165\\. _array[tp++] |= bb._array[sp] \u003e\u003e\u003e (NBits - _avail); 166\\. _array[tp] = bb._array[sp++] \u003c\u003c _avail; 167\\. }\n\n168\\. while (sp \u003c bb._free);\n\nThis can lead to an ArrayIndexOutOfBoundsException when the _free index for the incoming BitBuffer is outside the bounds of the target BitBuffer. We're not sure how the length of the file affects this result, but we did create the following test case to confirm:\n\n(1) Change the initial array size to 3 (instead of 256, to avoid the need for filling up one buffer with 256 words).\n(2) Define a static main method as follows:\n\npublic static void main (String[] args)\n{\n/* Instantiate both buffers. Note that both buffers can contain up to 96 bits (3 words * 32 bits per word) */\nBitBuffer target = new BitBuffer(), source = new BitBuffer();\n\n/* Add 8 bits to target – this changes the number of available bits and changes the _free index */\ntarget.append(0xAA, 8);\n\n/* Add 72 bits to source */\nsource.append(0x11223344, 32);\nsource.append(0x55667788, 32);\nsource.append(0x99, 8);\n\n/* Note here that target has enough room to consume all bits in source (72 + 8 \u003c 96),\nso the target buffer will not be resized in concatenate () */\n\n/* Flush outstanding bits in both buffers to array */\ntarget.close();\nsource.close();\n\n/* Invoke concatenate () and take note of the ArrayIndexOutOfBoundsException */\ntarget.concatenate(source);\n}\n\nWe believe the issue can be fixed by counting the number of remaining bits instead of watching the word pointer in the do-while loop:\n\n162\\. int tp = _free - 1; // target\n163\\. int sp = 0; // source\n164\\. int remaining = bb.bitCount(); // \u003c-- Begin with all bits in source remaining\n165\\. do {\n166\\. _array[tp++] |= bb._array[sp] \u003e\u003e\u003e (NBits - _avail);\n167\\. remaining = _avail; // \u003c- Deduct bits added to target from remaining\n168\\. if (remaining \u003e 0) // \u003c-- Are there bits remaining after the move?\n169\\.\n\n{ 170\\. _array[tp] = bb._array[sp++] \u003c\u003c _avail; 171\\. remaining -= NBits - _avail; // \u003c-- Deduct bits added to target 172\\. }\n\n173\\. }\n174\\. while (remaining \u003e 0); // Do this while there are bits remaining\n\nWe made this change and rebuilt the JavaHelp project, but making this change seems to have led to yet another exception:\n\njava.lang.ArrayIndexOutOfBoundsException: 17\nat com.sun.java.help.search.Decompressor.ascendingDecode(Decompressor.java:176)\nat com.sun.java.help.search.DocumentLists$MicroIndex.next(DocumentLists.java:105)\nat com.sun.java.help.search.DocumentLists$MicroIndex.openDocumentIndex(DocumentLists.java:137)\nat com.sun.java.help.search.DocumentLists$MicroIndex.\u003cinit\u003e(DocumentLists.java:79)\nat com.sun.java.help.search.DocumentLists.\u003cinit\u003e(DocumentLists.java:194)\nat com.sun.java.help.search.DocumentLists.invert(DocumentLists.java:270)\nat com.sun.java.help.search.DefaultIndexBuilder.close(DefaultIndexBuilder.java:130)\nat com.sun.java.help.search.Indexer.compile(Indexer.java:242)\nat com.sun.java.help.search.Indexer.main(Indexer.java:87)\n\nWith this exception, the index is not properly built, and so we rolled back our change in BitBuffer.java.\n\nIs there any way this issue can be fixed? Is there any design documentation that we might use to determine how this library works? The software code itself is very scarcely commented, making it very difficult to compare the intended functionality with the actual behavior.\n#### Environment\nn/a\n#### Affected Versions\n[current]","author":{"url":"https://github.com/glassfishrobot","@type":"Person","name":"glassfishrobot"},"datePublished":"2012-08-31T14:26:10.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/38/javahelp/issues/38"}