René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Seeking advice: remove is painfully slow","articleBody":"I'm finding that adding and searching an objectbox database is really fast. However, the remove operation is really slow (1 second per object.) The database is on a local NVME SSD drive. It contains about 20,000 hashes and takes about 6GB.\r\n\r\nMy find_unique hash_box.query operation is fast - it's literally the call to hash_box.remove that takes the time.\r\n\r\nWhat am I doing wrong?\r\n\r\n``` python\r\n@Entity()\r\nclass ImHash:\r\n    id = Id\r\n    key = String(index=Index(IndexType.HASH), unique=True)\r\n    cos_value = Float32Vector(index=HnswIndex(\r\n        dimensions=62720,\r\n        distance_type=VectorDistanceType.COSINE,\r\n    ))\r\n\r\n\r\ndef hash_image(im: Image.Image) -\u003e list[float]:\r\n    vector = img2vec.get_vec(im, tensor=True)\r\n    return vector.detach().cpu().numpy().flatten()\r\n\r\n\r\ndef hash_and_store(name_or_fp, key: str):\r\n    im = Image.open(name_or_fp)\r\n    h = hash_image(im)\r\n    ih = find_unique(key)\r\n    if ih is None:\r\n        # create\r\n        ih = ImHash()\r\n        ih.key = key\r\n    ih.cos_value = h\r\n    with store_lock:\r\n        hash_box.put(ih)\r\n\r\n\r\ndef init(db_dir: pathlib.Path):\r\n    global store, hash_box, img2vec\r\n    store = Store(directory=str(db_dir / directory_name),\r\n                  model_json_file=str(db_dir / json_model_name),\r\n                  max_db_size_in_kb=10 * 1024 * 1024)\r\n    hash_box = store.box(ImHash)\r\n    img2vec = Img2Vec(cuda=False, model='efficientnet_b0')\r\n\r\n\r\ndef close():\r\n    store.close()\r\n\r\n\r\ndef find_unique(key: str):\r\n    with store_lock:\r\n        query = hash_box.query(ImHash.key.equals(key)).build()\r\n        result = query.find()\r\n    if len(result) == 0:\r\n        return None\r\n    elif len(result) \u003e 1:\r\n        print('Multiple matches found')\r\n        return None\r\n    else:\r\n        return result[0]\r\n\r\n\r\ndef find_similar(key: str) -\u003e list[tuple[ImHash, float]]:\r\n    target = find_unique(key)\r\n    with store_lock:\r\n        query = hash_box.query(ImHash.cos_value.nearest_neighbor(target.cos_value, 8)).build()\r\n        results = query.find_with_scores()\r\n    results.sort(key=lambda x: x[1])\r\n    return results\r\n\r\n\r\ndef remove(key: str):\r\n    target = find_unique(key)\r\n    if target is not None:\r\n        with store_lock:\r\n            hash_box.remove(target)\r\n\r\n\r\ndef remove_many(keys: list[str]):\r\n    with store.write_tx():\r\n        for k in keys:\r\n            i = find_unique(k)\r\n            if i is None:\r\n                print('Hash key \"%s\" was already gone' % k)\r\n            else:\r\n                with store_lock:\r\n                    hash_box.remove(i.id)\r\n```","author":{"url":"https://github.com/patknight","@type":"Person","name":"patknight"},"datePublished":"2024-11-17T21:40:06.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/25/objectbox-python/issues/25"}