tianyu liu podcasts

ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code

Play Episode Listen Later Jul 8, 2024 27:24

Despite Large Language Models (LLMs) like GPT-4 achieving impressive results in function-level code generation, they struggle with repository-scale code understanding (e.g., coming up with the right arguments for calling routines), requiring a deeper comprehension of complex file interactions. Also, recently, people have developed LLM agents that attempt to interact with repository code (e.g., compiling and evaluating its execution), prompting the need to evaluate their performance. These gaps have motivated our development of ML-Bench, a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. Addressing the need for LLMs to interpret long code contexts and translate instructions into precise, executable scripts, ML-Bench encompasses annotated 9,641 examples across 18 GitHub repositories, challenging LLMs to accommodate user-specified arguments and documentation intricacies effectively. To evaluate both LLMs and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment. Our findings indicate that while GPT-4o leads with a Pass@5 rate surpassing 50%, there remains significant scope for improvement, highlighted by issues such as hallucinated outputs and difficulties with bash script generation. Notably, in the more demanding ML-Agent-Bench, GPT-4o achieves a 76.47% success rate, reflecting the efficacy of iterative action and feedback in complex task resolution. Our code, dataset, and models are available at https://github.com/gersteinlab/ML-bench. 2023: Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yan Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Zheng Li, Liang Chen, Yiming Zong, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, Mark B. Gerstein https://arxiv.org/pdf/2311.09835

ai pass code addressing tasks evaluating machine learning bench ml linux gpt github notably llm large language models repository tianyu liu

4. Tianyu Liu

Making Menuhin

Play Episode Listen Later May 22, 2020 14:43

His parents knew his potential as a violinist at an early age, even making him stop playing basketball for fear of him damaging his fingers. Tianyu Liu has followed through on his parents' belief, and calls on the technical approach of his home country of China, the focus on musicality from his time in America and the emotion of his audiences to inform his playing style. Hear him perform the third movement from Bach’s Solo Violin Sonata No. 2 in A Minor and the first movement from Eugene Ysaye’s Solo Violin Sonata No. 4. The 21-year old currently attends College of Charleston in South Carolina, and is a senior competitor in Menuhin Competition Richmond 2021. Visit vpm.org/menuhin for more information on the upcoming competition.

america china college south carolina charleston minor bach tianyu liu