Learn how to use LLaMA CPP (usually stylized as llama.cpp) to run a large language model (LLM) server on the Tufts high-performance compute (HPC) cluster. Access various open-weight models using a simple web-based chat interface, programmatically via an API, or from within existing AI tools and applications (also called harnesses) running off-cluster.
Session Outline
- What is local (self-hosted) AI and why does it matter?
- Architecture of the LLaMA CPP Server on the Tufts HPC
- Launching and configuring the server via Open OnDemand
- Overview of the curated predownloaded model catalog
- Requesting appropriate computational (GPU) resources
- Capabilities and constraints of the built-in chat interface
- API endpoints and querying the server programmatically
- Connecting the server to external AI tools and harnesses
Prerequisites
Interactive hands-on participation requires an active Tufts HPC cluster account. Submit a new cluster account request at least three business days in advance if needed.
Access to HPC resources is only possible from within the Tufts network. Off campus attendees should set up Cisco Secure Client and connect to the Tufts VPN beforehand.
Logistics
This is a virtual session offered via Zoom. Open to all Tufts affiliates. Advance registration required. Space limited to ensure an engaging experience. Waitlisted registrants admitted based on available capacity. Sessions with fewer than five registrations may be cancelled.