René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Models fail to load","articleBody":"I'm getting the following error in the final step of the quickstart:\r\n\r\n`unknown tensor 'transformer.h.0.attn.q_attn.weight' in model file`\r\n\r\nInput line: \r\n`./main -m models/bigcode/gpt_bigcode-santacoder-ggml.bin -p \"def fibonnaci(\" --top_k 0 --top_p 0.95 --temp 0.2`\r\n\r\nOutput:\r\n```\r\nmain: seed = 1687068338\r\nstarcoder_model_load: loading model from 'models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin'\r\nstarcoder_model_load: n_vocab = 49280\r\nstarcoder_model_load: n_ctx   = 2048\r\nstarcoder_model_load: n_embd  = 2048\r\nstarcoder_model_load: n_head  = 16\r\nstarcoder_model_load: n_layer = 24\r\nstarcoder_model_load: ftype   = 1003\r\nstarcoder_model_load: qntvr   = 1\r\nstarcoder_model_load: ggml ctx size = 1794.97 MB\r\nstarcoder_model_load: memory size =   768.00 MB, n_mem = 49152\r\nstarcoder_model_load: unknown tensor 'transformer.h.0.attn.q_attn.weight' in model file\r\nmain: failed to load model from 'models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin'\r\n```\r\n\r\nNotable differences from the sample output:\r\n- `starcoder_model_load: ftype   = 1` in my output vs `starcoder_model_load: ftype   = 3`\r\n(quanitzed models were produced with `./quantize models/bigcode/gpt_bigcode-santacoder-ggml.bin models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin 3`; non-quanitzed model fails with a similar error)\r\n- `starcoder_model_load: qntvr   = 1` in my output vs. no info on `qntvr` in the sample output\r\n\r\nOther notes:\r\n- this is running on a 2019 Intel MBP, not an M1\r\n- `conda list` is reproduced below in case I'm somehow missing a dependency\r\n```\r\n# Name                    Version                   Build  Channel\r\naccelerate                0.20.3             pyhd8ed1ab_0    conda-forge\r\nblas                      1.0                         mkl\r\nbrotlipy                  0.7.0           py310hca72f7f_1002\r\nbzip2                     1.0.8                h1de35cc_0\r\nca-certificates           2023.5.7             h8857fd0_0    conda-forge\r\ncertifi                   2023.5.7           pyhd8ed1ab_0    conda-forge\r\ncffi                      1.15.1          py310h6c40b1e_3\r\ncharset-normalizer        2.0.4              pyhd3eb1b0_0\r\nclick                     8.0.4           py310hecd8cb5_0\r\ncryptography              39.0.1          py310hf6deb26_2\r\ndataclasses               0.8                pyh6d0b6a4_7\r\nfilelock                  3.9.0           py310hecd8cb5_0\r\nfuture                    0.18.3          py310hecd8cb5_0\r\nhuggingface_hub           0.15.1                     py_0    huggingface\r\nidna                      3.4             py310hecd8cb5_0\r\nimportlib-metadata        6.0.0           py310hecd8cb5_0\r\nimportlib_metadata        6.0.0                hd3eb1b0_0\r\nintel-openmp              2023.1.0         ha357a0b_43547\r\njoblib                    1.2.0           py310hecd8cb5_0\r\nlibcxx                    14.0.6               h9765a3e_0\r\nlibffi                    3.4.4                hecd8cb5_0\r\nlibgfortran               5.0.0           11_3_0_hecd8cb5_28\r\nlibgfortran5              11.3.0              h9dfd629_28\r\nlibopenblas               0.3.21               h54e7dc3_0\r\nlibprotobuf               3.20.3               hfff2838_0\r\nlibuv                     1.44.2               h6c40b1e_0\r\nllvm-openmp               14.0.6               h0dcd299_0\r\nmkl                       2023.1.0         h59209a4_43558\r\nmkl-service               2.4.0           py310h6c40b1e_1\r\nmkl_fft                   1.3.6           py310h3ea8b11_1\r\nmkl_random                1.2.2           py310h3ea8b11_1\r\nncurses                   6.4                  hcec6c5f_0\r\nninja                     1.10.2               hecd8cb5_5\r\nninja-base                1.10.2               haf03e11_5\r\nnumpy                     1.24.3          py310h827a554_1\r\nnumpy-base                1.24.3          py310ha186be2_1\r\nopenssl                   3.1.1                h8a1eda9_1    conda-forge\r\npackaging                 23.0            py310hecd8cb5_0\r\npip                       23.1.2          py310hecd8cb5_0\r\nprotobuf                  3.20.3          py310hcec6c5f_0\r\npsutil                    5.9.5           py310h90acd4f_0    conda-forge\r\npycparser                 2.21               pyhd3eb1b0_0\r\npyopenssl                 23.0.0          py310hecd8cb5_0\r\npysocks                   1.7.1           py310hecd8cb5_0\r\npython                    3.10.11              h5ee71fb_3\r\npython_abi                3.10                    2_cp310    conda-forge\r\npytorch                   1.13.1          cpu_py310h9e40b02_0\r\npyyaml                    6.0             py310h6c40b1e_1\r\nreadline                  8.2                  hca72f7f_0\r\nregex                     2022.7.9        py310hca72f7f_0\r\nrequests                  2.29.0          py310hecd8cb5_0\r\nsacremoses                master                     py_0    huggingface\r\nsetuptools                67.8.0          py310hecd8cb5_0\r\nsix                       1.16.0             pyhd3eb1b0_1\r\nsqlite                    3.41.2               h6c40b1e_0\r\ntbb                       2021.8.0             ha357a0b_0\r\ntk                        8.6.12               h5d9f67b_0\r\ntokenizers                0.11.4          py310h8776b5c_1\r\ntqdm                      4.65.0          py310h20db666_0\r\ntransformers              4.28.1                     py_0    huggingface\r\ntyping-extensions         4.6.3           py310hecd8cb5_0\r\ntyping_extensions         4.6.3           py310hecd8cb5_0\r\ntzdata                    2023c                h04d1e81_0\r\nurllib3                   1.26.16         py310hecd8cb5_0\r\nwheel                     0.38.4          py310hecd8cb5_0\r\nxz                        5.4.2                h6c40b1e_0\r\nyaml                      0.2.5                haf1e3a3_0\r\nzipp                      3.11.0          py310hecd8cb5_0\r\nzlib                      1.2.13               h4dc903c_0\r\n```","author":{"url":"https://github.com/peter-ciccolo","@type":"Person","name":"peter-ciccolo"},"datePublished":"2023-06-18T06:08:49.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/21/starcoder.cpp/issues/21"}