BaseRT
BaseRT is a from-scratch, native Metal LLM inference runtime for Apple Silicon that delivers best-in-class prefill and decode throughput while shipping as a CLI, C API, and multi-language bindings with OpenAI-compatible serving.
https://www.basecompute.co/getbasert?ref=producthunt
Product Information
Updated:Jul 21, 2026
What is BaseRT
BaseRT is an AI inference runtime built specifically for running large language models on Apple Silicon. Unlike many runtimes that sit on top of general-purpose ML frameworks, BaseRT is written directly against Apple’s Metal GPU API and intentionally avoids dependencies on MLX, PyTorch, CoreML, or other intermediate layers. It is distributed as an easy-to-install toolchain (CLI plus a stable C API) with bindings for languages such as Python, Node, Rust, and Swift, enabling both local use (pull and chat with models) and deployment use cases (serving an OpenAI-compatible API).
Key Features of BaseRT
BaseRT is a high-performance LLM inference runtime for Apple Silicon built directly on Apple’s Metal GPU API (no MLX, PyTorch, CoreML, or other intermediate frameworks). It focuses on maximizing throughput and lowering overhead via chip-specific kernel fusion, unified-memory-aware optimizations, and custom dispatch logic, achieving best-in-class decode and prefill performance versus common Apple-local runtimes. BaseRT ships as a CLI plus a stable C API with language bindings, and can quickly pull models from Hugging Face, run local chat, or serve an OpenAI-compatible HTTP API for apps and agents—keeping inference private and on-device.
Native Metal runtime (framework-free): Implemented directly against Apple’s Metal API to avoid abstraction overhead from MLX/PyTorch/CoreML and better match Metal’s execution model and Apple Silicon’s unified memory topology.
Best-in-class throughput on Apple Silicon: Benchmarked as faster than MLX and llama.cpp, with reported gains up to ~33% on decode and large improvements on prefill (especially strong on long prompts and MoE-style workloads).
Architecture descriptor for model families: Encodes architectural differences (activations, normalization variants, attention/positional conventions, MoE routing specifics) in a compact descriptor rather than hard-coding many branches in the inference loop.
Broad quantization support: Supports multiple quantization formats (from low-bit quantization up to FP16), enabling users to trade off quality, memory, and speed across Apple M-series devices.
Developer-friendly CLI workflow: Install with a single script, pull models from Hugging Face and convert to a runtime format, then run chat or serving commands with minimal setup.
OpenAI-compatible local serving + bindings: Runs an OpenAI-compatible HTTP server for chat/completions and provides a stable C API with bindings (e.g., Python/Node/Rust/Swift) for embedding into apps and tools.
Use Cases of BaseRT
Private on-device assistants for enterprises: Run internal chat assistants locally on Apple Silicon (no data leaving the device), useful for regulated environments and sensitive documents.
Local coding agents and developer tooling: Serve a local model and point coding agents/IDEs at the OpenAI-compatible endpoint to get fast, low-latency code help without cloud API keys.
Edge inference for offline or low-latency apps: Deploy LLM features on laptops/tablets in field work, healthcare, or travel scenarios where connectivity is limited and latency matters.
Product prototyping and evaluation on Macs: Quickly test multiple open models and quantizations via CLI (pull/chat/serve) to benchmark UX, latency, and cost before committing to cloud deployment.
Embedded LLM capabilities in macOS/iOS apps: Use the C API and language bindings to integrate local LLM inference into native applications that need predictable performance and privacy.
Pros
High performance on Apple Silicon (notably strong decode and prefill throughput versus common local runtimes).
Framework-free Metal approach reduces overhead and improves hardware-specific optimization potential.
Convenient packaging: CLI + OpenAI-compatible server + stable C API with multiple language bindings.
Cons
Apple Silicon/Metal-focused (not a general cross-platform inference runtime).
Some competing approaches may leverage Apple Neural Engine via MPSGraph for certain workloads, while BaseRT’s path is described as GPU-focused (at least in current comparisons).
Requires model conversion to a runtime-specific format (e.g., pulled/converted to a BaseRT format) rather than running arbitrary checkpoints directly.
How to Use BaseRT
1) Install BaseRT (CLI + engine): On Apple Silicon macOS, install BaseRT onto your PATH by running:
curl -LsSf https://basecompute.co/install.sh | sh
After install, confirm the `basert` command is available (e.g., run `basert --help`).
2) Pull a model from Hugging Face (and convert to .base format): Use the BaseRT CLI to download a supported model from Hugging Face and convert it into BaseRT’s optimized `.base` format:
basert pull Qwen/Qwen3-4B
(Replace with any supported model ID.)
3) Chat with a local model from the terminal: Start an interactive chat session with a model you pulled:
basert chat Qwen/Qwen3-4B
This runs the model locally on your machine.
4) Serve an OpenAI-compatible HTTP API locally: Run BaseRT as a local server that exposes OpenAI-compatible endpoints:
basert serve Qwen/Qwen3-4B --api-key "$(uuidgen)" --port 8080
Use the printed/selected API key when calling the server from clients.
5) (Optional) Run a coding agent against your local BaseRT server: Serve a model and connect a local coding agent so requests stay on-device:
# Serve a model
basert serve basecompute/gemma-4-E4B-it
# Install the coding-agent plugin
pi install git:github.com/basecompute/pi-basert
# Run the agent
pi
BaseRT FAQs
BaseRT is an AI inference runtime from Base Compute designed to run large language models efficiently on Apple Silicon. It is written directly against Apple’s Metal API (not built on MLX, PyTorch, CoreML, or other intermediate frameworks) and is positioned as “the fastest LLM inference runtime for Apple Silicon.”
Popular Articles
Atoms: A Multi-Agent AI Platform That Transforms Ideas into Launch-Ready Products
May 22, 2026
Nano Banana SBTI: What It Is, How It Works, and How to Use It in 2026
Apr 15, 2026
Atoms Review — The AI Product Builder Redefining Digital Creation in 2026
Apr 10, 2026
Kilo Claw: How to Deploy and Use a True "Do‑It‑For‑You" AI Agent(2026 Update)
Apr 3, 2026
Latest AI Tools Similar to BaseRT
Athena AI
Athena AI is a versatile AI-powered platform offering personalized study assistance, business solutions, and life coaching through features like document analysis, quiz generation, flashcards, and interactive chat capabilities.
Aguru AI
Aguru AI is an on-premises software solution that provides comprehensive monitoring, security, and optimization tools for LLM-based applications with features like behavior tracking, anomaly detection, and performance optimization.
GOAT AI
GOAT AI is an AI-powered platform that provides one-click summarization capabilities for various content types including news articles, research papers, and videos, while also offering advanced AI agent orchestration for domain-specific tasks.
GiGOS
GiGOS is an AI platform that provides access to multiple advanced language models like Gemini, GPT-4, Claude, and Grok with an intuitive interface for users to interact with and compare different AI models.
Popular AI Tools Like BaseRT
GPT‑5.5 | ChatGPT Official
GPT‑5.5 in ChatGPT is OpenAI’s latest work-focused model designed to understand complex goals, use tools effectively, check its work, and carry multi-step tasks (coding, research, documents, spreadsheets) through to completion with stronger safeguards.
SearchGPT
SearchGPT is an AI-powered search prototype by OpenAI that provides fast, conversational answers with clear sources using GPT models.
ContextGem
ContextGem is a free, open-source LLM framework that simplifies structured data and insights extraction from documents with minimal code through powerful built-in abstractions and automated features.
AI CLI
AI CLI is an open-source command-line interface tool that brings AI capabilities directly to your terminal, allowing you to interact with various AI models like OpenAI's GPT and Anthropic's Claude through simple commands.