Fine Tuning

To do finetuning, first ensure you have these installed in the python environment:

• Unsloth
• Cuda versions of the Torch library
• Llama.cpp in the .unsloth folder.

Run train_dpo_gemma.py or train_sft_gemma.py to fine-tune. When fine-tuning completes, GGUF model files are created and saved under the fine_tuning folder. Use this with Ollama for inference.

Fine-tuned weights can also be found:

• Gemma 4 SFT: https://cometmail-my.sharepoint.com/:u:/g/personal/axd210123_utdallas_edu/IQCgYR9yq3ksS5LGFR6N2SzfAaqw4QjJHhkM-UtaFiibG8U?e=N25sdl
• Gemma 4 DPO: https://cometmail-my.sharepoint.com/:u:/g/personal/axd210123_utdallas_edu/IQBqICh_3oMwRK47STXLh8yPAWW_vwUw1mp9nluBOAz_I60?e=grPNIF

To Serve Models with Ollama and Run Benchmarks

To serve the models with Ollama, you must first create the model, and open the Ollama server to non localhost connections.

On Ollama

$env:OLLAMA_HOST="0.0.0.0"
$env:OLLAMA_ORIGINS="*"

ollama create <PUT MODEL NAME HERE> -f Modelfile
ollama list
ollama serve

Once served, the model will be available at PORT 11434 by default at the IP address of the computer running Ollama. Run ipconfig to get the IP address.

To run the benchmark, do the following on another computer.

Docker Init (On another machine)

docker run --name cweval --rm -it --net host `
  -v /var/run/docker.sock:/var/run/docker.sock `
  -v "${PWD}/evals:/home/ubuntu/CWEval/evals" `
  co1lin/cweval zsh

In Docker Instance

source .env

python cweval/commons.py compile_all_in --path benchmark/
echo $?

pytest benchmark/ -x -n 24

export OLLAMA_API_BASE="http://<IP_ADDRESS>:11434"

Generate

After the CWEval Docker environment is set up, the benchmark has 2 steps: generation and evaluation. During generation, the benchmark sends the coding tasks to the LLM endpoing and saves teh result. To do it, run the generate script. Below are examples for how the script can be used.

python cweval/generate.py gen --n 3 --temperature 0.8 --num_proc 16 --eval_path evals/eval_4omini_t8 --model gpt-4o-mini-2024-07-18

python3 cweval/generate.py gen --model ollama/gemma4 --num_proc 1 --n 1 --eval_path evals/gemma4_results

Evaluation

All the generated code is saved to the evals folder. To run evaluation, run the evaluation script. Examples are provided below. Note: you may need to run the sudo command before running the evaluate script if you are running evaluation also inside Docker (--docker True)

sudo chmod 666 /var/run/docker.sock

python cweval/evaluate.py pipeline --eval_path evals/dir --num_proc 8 --docker True

python3 cweval/evaluate.py eval --eval_path evals/gemma4_results

Additional Scripts

clean_evals.py can be run after running the Generation step of the benchmark to clean up the generated code files in case the model produced files with markdown syntax surrounding the files like ```python. Models sometimes don't produce explicitly just the code files - they might add explanations of the code before the code, so this can help fix that.

After evaluating a model, it prints out the results. Copy these results to a file called score.txt. This file should be in evals/model_name. Then, you can run compile_results.py to compile benchmark results into a CSV file.

Run plot_results.py or plot_bar.py after the CSV file is generated to get charts of the results.