An 8B-parameter model does not fit on an 8GB edge board in half precision, and it does not train on a free Colab T4 without 4-bit tricks either. This project works both ends of that squeeze: fine-tune ...

[2024/11/26] 📣 Introducing TensorRT LLM for Jetson AGX Orin, making it even easier to deploy on Jetson AGX Orin with initial support in JetPack 6.1 via the v0.12.0-jetson branch of the TensorRT LLM ...