Open Source AI Chat Robot MLC LLM Launched On GitHub

A new open-source project called MLC LLM has been launched on GitHub. It can run completely locally without networking, and even old computers with graphics cards and Apple iPhones can run.

The MLC LLM project introduction states: “MLC LLM is a general solution that allows any language model to be deployed locally on a diverse set of hardware backends and native applications, in addition to an efficient framework for everyone to further Optimize model performance for your own use case. Everything runs locally, without server support, and is accelerated by local GPUs on phones and laptops. Our mission is to empower everyone to develop, optimize, and deploy AI models locally on their devices. “

MLC LLM Project Official Demonstration

We checked the GitHub page and found that the developers of this project came from the Catalyst project of Carnegie Mellon University, the SAMPL machine learning research group, the University of Washington, Shanghai Jiaotong University, and OctoML. They also have a related project called Web LLM, which runs an AI chatbot entirely in a web browser.

MLC is the abbreviation of Machine Learning Compilation

MLC LLM uses Vicuna-7B-V1.1, which is a lightweight LLM based on Meta LLaMA. Although the effect is not as good as GPT3.5 or GPT4, it has an advantage in size. Currently, MLC LLM is available for Windows, Linux, macOS, and iOS platforms, and there is no version for Android yet.

According to tests by foreign media tomshardware, the Apple iPhone 14 Pro Max and iPhone 12 Pro Max phones with 6GB of memory successfully run MLC LLM with an installation size of 3GB. And the Apple iPhone 11 Pro Max with 4GB RAM cannot run MLC LLM.

Picture source tomshardware

MLC LLM was also tested successfully on the ThinkPad X1 Carbon (6th Gen), a notebook with an i7-8550U processor, no discrete graphics card, and an Intel UHD 620 GPU. MLC LLM needs to be run through the command line on the PC platform.