How to Build Your First Local CLI Agent Using Ollama & Python

TL;DR / Key Takeaways

A local CLI agent — running entirely on your machine via Ollama, with zero data leaving your network — is the right starting point for anyone building agentic systems that touch sensitive files, internal codebases, or regulated data. Key points:

• Ollama handles model serving; your Python code handles the agent loop. Ollama exposes a local OpenAI-compatible API (default http://localhost:11434), so the agent architecture is identical to a cloud-API agent — only the endpoint changes.
• Tool execution is the dangerous part, not the LLM call. File read/write and shell execution tools must be sandboxed with explicit allowlists — never let the model construct and run arbitrary shell strings unchecked.
• Local models (Llama 3, DeepSeek) are weaker at structured tool-calling than frontier hosted models — expect to invest more in prompt scaffolding and output validation/retry logic than you would with a hosted frontier model.
• The ReAct loop pattern (Reason → Act → Observe) is the simplest reliable architecture for a local CLI agent — you don’t need a full framework (LangGraph/CrewAI) for a single-agent local tool.
• Nothing here requires internet access after the initial ollama pull — this is the core value proposition for teams with strict data residency or air-gapped requirements.

---

Core Concept: Why Build Local Instead of Calling a Cloud API?

The architecture of an agent — LLM reasons about a task, decides to call a tool, observes the result, repeats until done — doesn’t change based on where the model runs. What changes is the trust boundary. A cloud-API agent sends your file contents, shell output, and reasoning traces to a third party on every call. A local agent, running Llama 3 or DeepSeek through Ollama on your own hardware, never does.

This matters specifically for:

• Codebases under NDA or containing proprietary logic — a local agent that reads and reasons about your source files never transmits them externally.
• Regulated data environments (healthcare, finance, government) where sending data to a third-party API is a compliance non-starter regardless of the provider’s data retention policy.
• Air-gapped or low-connectivity environments — once the model is pulled, the agent runs with zero network dependency.

The trade-off is capability: local open-weight models, particularly at 8B-13B parameter sizes runnable on consumer hardware, are meaningfully weaker at complex multi-step reasoning and reliable structured output than frontier hosted models. The architecture below compensates for this with tighter scaffolding.

---

Practical Tutorial: Building the Agent

Step 1 — Install Ollama and Pull a Model

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model with reasonable tool-use capability at a runnable size
ollama pull llama3.1:8b
# or, for stronger reasoning at higher VRAM cost:
ollama pull deepseek-r1:14b

Verify the local API is live:

curl http://localhost:11434/api/tags

Step 2 — Define the Tool Set (Sandboxed)

This is the section that determines whether your agent is safe to run. Never let the model generate a raw shell string that gets passed to subprocess.run(shell=True) — always route through an explicit, validated tool interface.

import subprocess
import os
from pathlib import Path

# Restrict all file operations to a single sandboxed working directory
SANDBOX_ROOT = Path("./agent_workspace").resolve()
SANDBOX_ROOT.mkdir(exist_ok=True)

# Explicit allowlist of shell commands the agent may invoke —
# never allow arbitrary command strings
ALLOWED_COMMANDS = {"ls", "cat", "grep", "wc", "find", "git"}

def _resolve_safe_path(relative_path: str) -> Path:
    """Resolve a path and refuse anything that escapes the sandbox."""
    target = (SANDBOX_ROOT / relative_path).resolve()
    if not str(target).startswith(str(SANDBOX_ROOT)):
        raise PermissionError(f"Path escapes sandbox: {relative_path}")
    return target

def read_file(relative_path: str) -> str:
    path = _resolve_safe_path(relative_path)
    if not path.exists():
        return f"Error: file not found: {relative_path}"
    return path.read_text(errors="replace")[:10_000]  # cap read size

def write_file(relative_path: str, content: str) -> str:
    path = _resolve_safe_path(relative_path)
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(content)
    return f"Wrote {len(content)} characters to {relative_path}"

def run_shell_command(command: str) -> str:
    parts = command.strip().split()
    if not parts or parts[0] not in ALLOWED_COMMANDS:
        return f"Error: command '{parts[0] if parts else ''}' is not in the allowlist"
    try:
        result = subprocess.run(
            parts,
            cwd=SANDBOX_ROOT,
            capture_output=True,
            text=True,
            timeout=10,          # hard timeout — never let a tool call hang
            shell=False,         # never shell=True with model-generated input
        )
        return result.stdout[:5_000] or result.stderr[:5_000]
    except subprocess.TimeoutExpired:
        return "Error: command timed out"

TOOLS = {
    "read_file": read_file,
    "write_file": write_file,
    "run_shell_command": run_shell_command,
}

Three non-negotiable safety patterns are baked in here: path sandboxing (no reads/writes outside agent_workspace/), a command allowlist (no arbitrary shell strings, shell=False always), and a hard timeout on every execution.

Step 3 — The Agent Loop (ReAct Pattern)

import json
import requests

OLLAMA_URL = "http://localhost:11434/api/chat"
MODEL = "llama3.1:8b"

SYSTEM_PROMPT = """You are a local CLI coding assistant with file and shell access.
On each turn, respond with EXACTLY one JSON object, no other text:

{"thought": "your reasoning", "action": "tool_name", "action_input": {...}}

or, when the task is complete:

{"thought": "your reasoning", "final_answer": "your response to the user"}

Available tools:
- read_file(relative_path: str)
- write_file(relative_path: str, content: str)
- run_shell_command(command: str)  # allowed commands: ls, cat, grep, wc, find, git
"""

def call_ollama(messages: list) -> str:
    response = requests.post(OLLAMA_URL, json={
        "model": MODEL,
        "messages": messages,
        "stream": False,
        "format": "json",  # forces valid JSON output where the model supports it
    })
    return response.json()["message"]["content"]

def run_agent(user_task: str, max_steps: int = 8):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_task},
    ]

    for step in range(max_steps):
        raw = call_ollama(messages)
        try:
            parsed = json.loads(raw)
        except json.JSONDecodeError:
            # Local models occasionally produce malformed JSON — retry once
            # with an explicit correction instruction rather than crashing.
            messages.append({"role": "assistant", "content": raw})
            messages.append({"role": "user", "content": "That was not valid JSON. Respond with valid JSON only."})
            continue

        print(f"[Step {step}] Thought: {parsed.get('thought')}")

        if "final_answer" in parsed:
            return parsed["final_answer"]

        action = parsed.get("action")
        action_input = parsed.get("action_input", {})

        if action not in TOOLS:
            observation = f"Error: unknown tool '{action}'"
        else:
            observation = TOOLS[action](**action_input)

        messages.append({"role": "assistant", "content": raw})
        messages.append({"role": "user", "content": f"Observation: {observation}"})

    return "Max steps reached without a final answer."

if __name__ == "__main__":
    task = input("Task: ")
    print(run_agent(task))

Step 4 — Harden Against Malformed Tool Calls

Local models, especially at smaller parameter counts, are meaningfully less reliable at strict JSON adherence than frontier hosted models. Two additions make this production-viable rather than a fragile demo:

def validate_action_input(action: str, action_input: dict) -> tuple[bool, str]:
    """Reject malformed or suspicious tool calls before execution."""
    if action == "write_file":
        if "relative_path" not in action_input or ".." in action_input.get("relative_path", ""):
            return False, "Invalid or unsafe relative_path"
    if action == "run_shell_command":
        cmd = action_input.get("command", "")
        if any(bad in cmd for bad in [";", "&&", "|", ">", "`", "$("]):
            return False, "Command contains disallowed shell metacharacters"
    return True, ""

Reject-and-retry (feeding the validation error back to the model as an observation) is far more robust than trying to make the model “always” produce safe input — local models will occasionally attempt shell chaining or path traversal, not out of malice, but because smaller models are simply less reliable at following constraints under complex prompts.

---

Tool / Solution Comparison Table

Approach	Data Privacy	Reasoning Quality	Hardware Requirement	Setup Effort
Ollama + Llama 3.1 8B (local)	Fully local, zero external calls	Moderate — sufficient for well-scaffolded tool use	Runs on most modern laptops (16GB+ RAM recommended)	Low
Ollama + DeepSeek-R1 14B (local)	Fully local	Stronger multi-step reasoning, higher VRAM cost	Needs a dedicated GPU or 32GB+ RAM for reasonable speed	Low-Medium
Cloud API agent (Claude/GPT)	Data leaves the local machine	Highest — best structured tool-calling reliability	None (network only)	Low
Hybrid (local for sensitive steps, cloud for complex reasoning)	Selective — only non-sensitive steps leave the machine	Best of both, more architecture complexity	Both local and network	Medium-High

---

Actionable Checklist / Next Steps

• Install Ollama and pull a model sized to your hardware (llama3.1:8b for most laptops, larger models for GPU-equipped machines).
• Build every tool with explicit sandboxing — path resolution checks, command allowlists, and hard timeouts, before writing a single line of agent-loop code.
• Never use subprocess.run(shell=True) with model-generated input — always parse into an argument list and validate against an allowlist.
• Add JSON validation and a retry path for malformed model output — local models fail structured output more often than hosted frontier models.
• Cap read/output sizes on every tool to avoid flooding the context window with a single large file or command output.
• Test the agent against adversarial-style inputs (a task designed to make it attempt path traversal or command chaining) before trusting it with real file access.

---

Next in this series: Text-to-Video Prompting Mastery: 15 Camera Movements That Actually Work — Day 07 shifts back to AI media production workflows.