AI4JVM — Java AI Ecosystem Guide: Agent Frameworks, Inference Engines & Tools

Latest Headlines

Spring AI 2.0.0-M3, 1.1.3, and 1.0.4 Available — MCP annotations moved into core, 23 new features, plus maintenance releases with bug fixes
Introducing Tracy: The AI Observability Library for Kotlin — production-grade AI observability for Kotlin apps, debug failures and track LLM usage with OpenTelemetry
Multi-Language MCP Server Performance Benchmark — comparative analysis across Go, Java, Python, and TypeScript
2026 Java AI Apps — building AI applications with Java in 2026
Agent Memory Is Not a Greenfield Problem — ground it in your existing data
Production LangChain4j at Devoxx Belgium — advanced RAG, agentic workflows, and production tips

Agent Frameworks & Libraries

Framework

Spring AI

The Spring ecosystem's official AI framework. Portable abstractions across 20+ model providers, tool calling, RAG, chat memory, vector stores, and MCP support. Built by the Spring team at Broadcom.

Framework

The most popular Java LLM library. Unified API across 20+ LLM providers and 30+ embedding stores. Three levels of abstraction from low-level prompts to high-level AI Services. Supports RAG, tool calling, MCP, and agents.

Framework

Embabel

Created by Rod Johnson (Spring Framework creator). JVM agent framework using Goal-Oriented Action Planning (GOAP) for dynamic replanning. Strongly typed, Spring-integrated, MCP support. Written in Kotlin with full Java interop.

Framework

Google ADK for Java

Google's Agent Development Kit — code-first Java toolkit for building, evaluating, and deploying AI agents. Supports Gemini natively plus third-party models via LangChain4j integration. A2A protocol for agent-to-agent communication.

Framework

Quarkus LangChain4j

Enterprise-grade Quarkus extension for LangChain4j. Native compilation with GraalVM, built-in observability (metrics, tracing, auditing), and Dev UI tooling. Maintained by Red Hat & IBM.

Framework

LangGraph4j

Build stateful, multi-agent applications with cyclical graphs. Inspired by Python's LangGraph, works with both LangChain4j and Spring AI. Persistent checkpoints, deep agent architectures, and a Studio web UI.

Framework

Koog (JetBrains)

Kotlin-native agent framework from JetBrains. Type-safe DSL, multiplatform (JVM, JS, WasmJS, Android, iOS), A2A protocol support, fault tolerance with persistence, and multi-LLM support.

Framework

Semantic Kernel (Java)

Microsoft's AI orchestration SDK with Java support. Merged with AutoGen into a unified Microsoft Agent Framework with deep Azure integration. Supports prompt chaining, planning, and memory.

SDK

MCP Java SDK

The official Java SDK for Model Context Protocol servers and clients. Co-maintained by the Spring AI team and Anthropic. Sync/async, STDIO/SSE/Streamable HTTP transports, OAuth support.

SDK

Anthropic Java SDK

Official Java SDK for the Claude Messages API. Streaming, retries, structured outputs, extended thinking, code execution, and files API. Build Java apps powered by Claude.

SDK

GitHub Copilot SDK for Java

Official Java SDK for embedding the GitHub Copilot agentic engine directly into Java applications. Uses the same agentic harness that powers the Copilot CLI — exposes planning, tool calling, file editing, and MCP integration via a simple Java API. Currently in technical preview.

Library

Tracy (JetBrains)

AI tracing library for Kotlin and Java. Captures structured traces from LLM interactions — messages, cost, token usage, and execution time. Implements OpenTelemetry Generative AI Semantic Conventions with exports to Langfuse, Weights & Biases, and more.

Library

Docling Java

Docling Java is the official Java client and tooling for Docling — a suite that converts messy documents into structured data and simplifies downstream document and AI processing by detecting tables, formulas, reading order, OCR, and much more.

Java with Code Assistants

Technologies that supercharge Java development when paired with AI code assistants — from MCP servers that give agents live Javadoc access, to reusable skill packages and IDE integrations.

MCP Server

Javadocs.dev MCP Server

Gives AI assistants live access to Java, Kotlin, and Scala library documentation from Maven Central. Six tools including latest-version lookup, Javadoc symbol browsing, and source file retrieval. Connect any MCP client via Streamable HTTP.

Assistant

JetBrains AI

AI-powered coding assistance built into IntelliJ IDEA and all JetBrains IDEs. Context-aware code completion, next-edit suggestions, and an agent-mode chat for refactoring, test generation, and complex tasks. Deep understanding of Java, Kotlin, and Scala project conventions. Supports cloud LLMs (Gemini, OpenAI, Anthropic) plus bring-your-own-key.

Skills

SkillsJars

A packaging format and registry for distributing reusable AI agent skills as Maven/Gradle JARs. Skills are Markdown files (SKILL.md) under META-INF/skills/ that teach AI agents domain-specific patterns. Discover and load skills on demand in Claude Code, Kiro, and Spring AI apps.

Inference & Training

Run models, train classifiers, and do ML inference directly on the JVM — no Python required.

Inference

Jlama

Modern LLM inference engine written in pure Java. Runs Llama, Gemma, Mistral, and more locally on CPU. Uses Java's Vector API (Project Panama) for SIMD-accelerated matrix math. Supports GGUF and SafeTensors formats, quantized models, and distributed inference.

Inference

Deep Java Library (DJL)

AWS's high-level, engine-agnostic deep learning framework. Supports PyTorch, TensorFlow, and MXNet backends. Used in production at Netflix and Amazon for real-time inference. DJLServing provides high-performance model serving.

Inference

ONNX Runtime Java

Run transformer and classical ML models directly on the JVM. Hardware acceleration via CUDA, ROCm, DirectML, and more. Enables deploying scikit-learn, PyTorch, and HuggingFace models in Java without Python or REST wrappers.

Training

Tribuo

Oracle Labs' ML library for classification, regression, clustering, and anomaly detection. Strong typing, provenance tracking for reproducibility, and integrations with XGBoost, ONNX Runtime, TensorFlow, and LibSVM.

Inference

GPULlama3.java

First Java-native Llama 3 implementation with automatic GPU acceleration via TornadoVM. No CUDA or native code needed — GPU-accelerated LLM inference in pure Java. From the University of Manchester's Beehive Lab.

Training

TensorFlow Java

Official Java bindings for TensorFlow. Train and deploy TF models entirely in Java. Used by Tribuo under the hood. Suitable for teams that want to stay within the JVM ecosystem while using TensorFlow's model formats.