René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Creating a Matrix with more than 2,147,483,647 elements","articleBody":"Hi,\nThe matrix dimensions in BidMat use the Int type, which seems to be limiting the number of elements I can fit in a matrix. After 2147483647 elements there are calculations in the Matrix class about the total number of elements that overflow and return a java.lang.NegativeArraySizeException.\n\nAs a background, my dataset is at most 15 billion numbers, as I am doing a PCA of ~ 1 million rows x 15,000 columns.. My use case is comparing the speed of [this implementation](https://blog.dominodatalab.com/pca-on-very-large-neuroimaging-datasets-using-pyspark/) to randomized SVD in BidMach/Mat and randomized SVD in numpy.\nWith 1.5 billion elements BidMach performed excellently (10s for the BidMach SVD with dim=50 on a beefy machine) but I cannot go further with it because I can't fit any more values in my matrices.\n\nCheers,\nAmedee\n","author":{"url":"https://github.com/amedeedaboville","@type":"Person","name":"amedeedaboville"},"datePublished":"2016-09-04T15:49:44.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/71/BIDMat/issues/71"}