Maciej Urbański

Work · AI_devs · Python + TypeScript

AI_devs — agent loops, RAG, evals, and MCP in practice

Two finished credentials, backed by code: AI_devs 4 Builders (2026) and AI_devs 3 Agents (2025). The Builders repo is a Python agent framework plus solved tasks. The same stack shows up in the TypeScript harness: tool loops, MCP, retrieval, traces, and evals.

My role Course work I implemented and kept: framework, task agents, and the retrieval/eval pieces — not a certificate-only claim.

Stack Python · TypeScript · MCP adapter · SQLite FTS5 + sqlite-vec · Neo4j · Langfuse · Zod / JSON schema

Context
The credentials are the public check. The repos are the proof: a framework with Tool, Agent, Context, an MCP-to-tool adapter, and run logs, then task agents that had to call tools, verify, and stop. Later lessons are retrieval and evals, which is the part an AI/ML role actually hires for.
Intervention

Built the learning framework, then used it — and the TypeScript harness — on the parts of the curriculum that are the stack: tools, MCP, RAG, graphs, traces, evals.

Tool loops and MCP

Python Agent runs a tool loop with JSON-schema tools, structured run logs, and an MCP adapter that exposes server tools as framework tools. Task agents include multi-step API calls, a verify loop, and confirmation before a side effect. AI_devs 3 covered tool-using assistants, Langfuse, and Neo4j.

Retrieval

Agentic RAG with multi-step retrieval. Hybrid RAG over SQLite FTS5 and sqlite-vec. Chunking strategies and an embedding similarity matrix. Graph RAG on Neo4j: hybrid search plus entity exploration. That is the ML side of the work: embeddings, similarity, and how retrieval is composed, not model training.

Evals, traces, sandbox

Langfuse at the adapter boundary. A synthetic tool-use eval and a response-correctness eval. Sandboxed code execution (QuickJS / Deno) with MCP file tools. Observational memory as an observer/reflector pair. A separate Python stdio MCP, transcribe_video, runs extract-audio then Whisper and returns markdown to the model.

Risks & how they were handled
  • Treating course demos as production → the CV and this page say which systems are live (Wrike, Halo reads/writes) and which are the harness and coursework.
  • Retrieval without a check → verify loops and tool-use evals sit next to the RAG lessons.
  • Tool side effects → confirmation gates in the course agent and the same confirm: true pattern on Halo.
What I'd do differently
  • Lift two or three Langfuse datasets out of the course tasks and point them at the Halo tool schemas.
  • Keep one hybrid index (FTS + vectors) as a reusable library instead of a lesson folder.
Outcome
The credential is public. Behind it: a Python agent framework with an MCP adapter, task agents that plan and verify, hybrid and graph RAG, embedding and chunking work, Langfuse traces and tool-use evals, and a Whisper MCP for speech-to-markdown. Credentials: AI_devs 4 Builders (2026-10-01), AI_devs 3 Agents (2025-03-03).

System map

Task / query

Agent tool loop

Python / TS

MCP adapter

tools

Framework

Hybrid + graph RAG

FTS · vectors · Neo4j

Eval traces

Langfuse

What it does

Solves AI_devs tasks with a reusable loop

Retrieves with FTS, vectors, or graph

Scores tool use with evals

Agent loops, retrieval, and evals from the AI_devs builds — one framework, many tasks.

What shipped

  • Agent framework Reusable tool loop and MCP adapter with JSON run logs — Python and TypeScript.
  • Hybrid and graph RAG Full-text, vector, and graph retrieval — SQLite FTS5 + sqlite-vec, Neo4j, embeddings, Whisper MCP.
  • Credentials AI_devs 4 Builders and AI_devs 3 Agents — linked from the case study.

Stack Python TypeScript MCP SQLite FTS5 sqlite-vec Neo4j Langfuse

← Agent harness Next: Wrike MCP →