LLM Gateway for Local AI

Run a secure, professional AI API proxy in front of your local Gemma 4 (or any Ollama model) on your Mac Mini

What is an LLM Gateway?

An LLM Gateway (also called AI API Proxy) sits between your agents/tools and your local LLM server (Ollama, MLX, llama.cpp, etc.).

It gives you clean virtual API keys, usage tracking, rate limits, budgets, and a nice admin dashboard — without ever exposing your raw Ollama port (11434) to the internet or other machines.

Why You Need This

Secure API key management (one key per agent, team member, or project)
Full OpenAI-compatible endpoint — works with every existing tool and agent
Usage monitoring, spend tracking, and rate limiting
Professional setup instead of pointing everything at http://your-mac:11434
Easy to brand later as llm.shadstone.com

Recommended Solution: LiteLLM Proxy

The fastest and most popular open-source LLM gateway in 2026. Perfect for running on your Apple Silicon Mac Mini.

1. Install LiteLLM

pip install 'litellm[proxy]'

2. Create config.yaml

model_list:
  - model_name: gemma4
    litellm_params:
      model: ollama/gemma4
      api_base: http://localhost:11434

3. Start the Proxy

litellm --config config.yaml --port 4000 --master-key sk-1234567890abcdef

4. Access Dashboard

Open http://localhost:4000/ui

Any agent now points to http://your-mac-ip:4000 with the new virtual key instead of raw Ollama.

Even Better Setups

OpenWebUI + LiteLLM

Beautiful ChatGPT-style UI + secure proxy. Both run together on the same Mac Mini via Docker.

llm-gateway (mylxsw)

Modern Next.js + shadcn/ui dashboard — perfect if you want to heavily brand it as part of Shadstone.

Fully Custom Build

Next.js 16 frontend + FastAPI backend + SQLite. Only needed for deep custom branding and features.

My Recommendation

Start with LiteLLM Proxy today — you’ll have a clean, secure, production-ready gateway in under 15 minutes.

Once you love the workflow, you can layer OpenWebUI for a nicer chat interface or build a custom shadcn-powered dashboard later.

Next Steps

Run the LiteLLM setup above on your Mac Mini
Test with your existing agents using the new endpoint
Secure remote access with Tailscale, ngrok, or Cloudflare Tunnel
Want the full Docker Compose version or MLX-LM config? Let me know and I’ll provide the exact files.