LLMs cannot reliably identify and reason about security vulnerabilities (yet?): a comprehensive evaluation, framework, and benchmarks

Ullah, SaadHan, MingjiPujar, SaurabhPearce, HammondCoskun, AyseStringhini, Gianluca2025-03-182025-03-182024-05-19S. Ullah, M. Han, S. Pujar, H. Pearce, A. Coskun, G. Stringhini. 2024. "LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks" 2024 IEEE Symposium on Security and Privacy (SP), pp.862-880. https://doi.org/10.1109/sp54263.2024.00210https://hdl.handle.net/2144/49910pp. 862-880LLMs cannot reliably identify and reason about security vulnerabilities (yet?): a comprehensive evaluation, framework, and benchmarksConference materials10.1109/sp54263.2024.002101021412