Precision | Language | Average | |||
---|---|---|---|---|---|
Java | JS | Python | Go | ||
k=1 | |||||
Chronos | 0.516 | 0.447 | 0.55 | 0.71 | 0.55575 |
VulLibMiner | 0.669 | 0.742 | 0.825 | 0.647 | 0.72075 |
GPT4 | 0.783 | 0.768 | 0.868 | 0.712 | 0.78275 |
Vicuna-13B | 0.71 | 0.776 | 0.935 | 0.804 | 0.80625 |
k=2 | |||||
Chronos | 0.673 | 0.598 | 0.554 | 0.767 | 0.648 |
VulLibMiner | 0.695 | 0.725 | 0.789 | 0.669 | 0.7195 |
GPT4 | 0.762 | 0.752 | 0.851 | 0.692 | 0.76425 |
Vicuna-13B | 0.722 | 0.779 | 0.858 | 0.747 | 0.7765 |
k=3 | |||||
Chronos | 0.741 | 0.648 | 0.569 | 0.781 | 0.68475 |
VulLibMiner | 0.724 | 0.723 | 0.782 | 0.715 | 0.736 |
GPT4 | 0.758 | 0.749 | 0.754 | 0.678 | 0.73475 |
Vicuna-13B | 0.743 | 0.776 | 0.841 | 0.785 | 0.78625 |
Recall | Language | ||||
---|---|---|---|---|---|
Java | JS | Python | Go | Average | |
k=1 | |||||
Chronos | 0.4 | 0.412 | 0.286 | 0.605 | 0.42575 |
VulLibMiner | 0.52 | 0.709 | 0.499 | 0.544 | 0.568 |
GPT4 | 0.596 | 0.714 | 0.542 | 0.58 | 0.608 |
Vicuna-13B | 0.552 | 0.736 | 0.621 | 0.688 | 0.64925 |
k=2 | |||||
Chronos | 0.623 | 0.591 | 0.346 | 0.778 | 0.5845 |
VulLibMiner | 0.647 | 0.719 | 0.561 | 0.653 | 0.645 |
GPT4 | 0.705 | 0.744 | 0.585 | 0.675 | 0.67725 |
Vicuna-13B | 0.669 | 0.771 | 0.622 | 0.733 | 0.69875 |
k=3 | |||||
Chronos | 0.722 | 0.645 | 0.392 | 0.605 | 0.591 |
VulLibMiner | 0.705 | 0.72 | 0.603 | 0.713 | 0.68525 |
GPT4 | 0.737 | 0.745 | 0.597 | 0.675 | 0.6885 |
Vicuna-13B | 0.72 | 0.773 | 0.657 | 0.782 | 0.733 |
F1 | Language | ||||
---|---|---|---|---|---|
Java | JS | Python | Go | ||
k=1 | |||||
Chronos | 0.451 | 0.429 | 0.376 | 0.653 | 0.482 |
VulLibMiner | 0.585 | 0.725 | 0.622 | 0.591 | 0.635 |
GPT4 | 0.677 | 0.74 | 0.667 | 0.639 | 0.684 |
Vicuna-13B | 0.621 | 0.755 | 0.746 | 0.741 | 0.719 |
k=2 | |||||
Chronos | 0.647 | 0.594 | 0.426 | 0.772 | 0.615 |
VulLibMiner | 0.67 | 0.722 | 0.656 | 0.661 | 0.68 |
GPT4 | 0.732 | 0.748 | 0.693 | 0.683 | 0.718 |
Vicuna-13B | 0.694 | 0.775 | 0.721 | 0.74 | 0.736 |
k=3 | |||||
Chronos | 0.731 | 0.646 | 0.464 | 0.682 | 0.634 |
VulLibMiner | 0.714 | 0.721 | 0.681 | 0.714 | 0.71 |
GPT4 | 0.747 | 0.747 | 0.666 | 0.676 | 0.711 |
Vicuna-13B | 0.731 | 0.774 | 0.738 | 0.783 | 0.759 |