René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Large latency on Tensor allocation","articleBody":"\r\n**System information**\r\n- TensorFlow version: 0.3.1\r\n- Doc Link: https://www.tensorflow.org/jvm/install\r\n\r\n\r\n**Describe the documentation issue**\r\n\r\nNot sure this is the correct forum but I would like to some guidance on how to setup sessions and resource management would be interesting. \r\n\r\nAfter two weeks trying to understand why latencies in 0.3.1 were completely uncontrollable (as compared to official 1.3.1) I ran into #208. This matches my observations. \r\n\r\nWe are trying to run Prediction on models with thousands of data points in different tensor per prediction. Memory allocation on the threads are in 20MB/s and there seems to be a sync between JavaCCP Allocation thread and our worker threads. In addition to this allocation using Size(1) tensor seems to be very slow (in the 7ms range). \r\n\r\nAfter reading #208, it seems we are doing everything wrong. But I don't really have a clear picture of how it should be done: Would `EagerSession` help? Could I use a `Session` per HTTP request? Should I allocate larger multi-dimensional  tensors instead of a single one? How should I configure thread pools?  I understand that the API is work in progress, but current documentation is very light on this kind of documentation.\r\n\r\nI don't think this is a bug, but I can convert into some other sort of issue. ","author":{"url":"https://github.com/nebulorum","@type":"Person","name":"nebulorum"},"datePublished":"2021-05-03T19:44:10.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":43},"url":"https://github.com/313/java/issues/313"}