René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Reproducing StarCoder2-Instruct","articleBody":"I am trying to recreate the `StarCoder2-Instruct-v0.1` model; however, the model produced by the provided command  in the README (copied below) does not match the evaluation of the `StarCoder2-Instruct-v0.1` model on HF. \r\n\r\nI actually see quite a bit of discrepancy between the two models' evaluations: `humaneval` on your HF version is 7 points higher than on my reproduced model (both models were evaluated locally by me in the same environment).\r\n\r\n```bash\r\nMODEL_KEY=bigcode/starcoder2-15b\r\nLR=1e-5\r\nEPOCH=4\r\nSEQ_LEN=1280\r\nWARMUP_RATIO=0.05\r\nOUTPUT_DIR=/path/to/output_model\r\nDATASET_FILE=/path/to/50k-dataset.jsonl\r\naccelerate launch -m star_align.train \\\r\n    --model_key $MODEL_KEY \\\r\n    --model_name_or_path $MODEL_KEY \\\r\n    --use_flash_attention True \\\r\n    --datafile_paths $DATASET_FILE \\\r\n    --output_dir $OUTPUT_DIR \\\r\n    --bf16 True \\\r\n    --num_train_epochs $EPOCH \\\r\n    --max_training_seq_length $SEQ_LEN \\\r\n    --pad_to_max_length False \\\r\n    --per_device_train_batch_size 1 \\\r\n    --gradient_accumulation_steps 64 \\\r\n    --group_by_length False \\\r\n    --ddp_find_unused_parameters False \\\r\n    --logging_steps 1 \\\r\n    --log_level info \\\r\n    --optim adafactor \\\r\n    --max_grad_norm -1 \\\r\n    --warmup_ratio $WARMUP_RATIO \\\r\n    --learning_rate $LR \\\r\n    --lr_scheduler_type linear\r\n```\r\n\r\nAre the parameters in the README correct for the released model? Are you adding anything in your `accelerate` config? i.e. any model wrappers or something else?\r\n\r\nFor the data, I just ran:\r\n```python\r\n\u003e\u003e\u003e from datasets import load_dataset\r\n\u003e\u003e\u003e load_dataset(\"bigcode/self-oss-instruct-sc2-exec-filter-50k\", split=\"train\").to_json(\"/path/to/50k-dataset.jsonl\", lines=True)\r\n```\r\n\r\nDo you have any ideas on how I can reproduce your model? Thanks!","author":{"url":"https://github.com/mstallone","@type":"Person","name":"mstallone"},"datePublished":"2024-05-04T20:36:28.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":5},"url":"https://github.com/6/selfcodealign/issues/6"}