LLM Orchestration

The Core API acts as an advanced orchestration layer between our user interfaces (Voice and Web) and Large Language Models (LLMs). Rather than simply proxying messages back and forth, it actively manages context, classifies intent, and injects dynamic state.

Rendez-vous.ai is built to be model-agnostic, supporting both Google Gemini and OpenAI depending on the specific agent’s configuration and requirements.

The Supervisor Pattern (Deprecated)

Before a user’s message is sent to the primary conversational LLM, it often passes through a Supervisor.

Located at src/services/llm/supervisor.ts, the Supervisor is an AI classification model responsible for determining whether the user’s input requires the invocation of an external tool.

By separating tool classification from conversational generation, we achieve higher accuracy and lower latency.

• Model Used: The Supervisor specifically utilizes Google’s lightweight gemini-3.1-flash-lite-preview model. It is heavily optimized for speed and structured outputs.
• How it works: It takes the agent’s systemInstructions, the list of tools assigned to the agent, and the raw input. It returns a strict JSON object dictating if a tool should be used (toolUse: boolean) and, if so, which tools (toolName: string[]).

// A simplified view of the Supervisor's output schema
const schema = z.object({
  toolUse: z.boolean(),
  toolName: z.array(z.string()).optional(),
});

Dynamic Prompt Variables

Agents configured in the Dashboard (via Payload CMS) have static system prompts. However, to effectively book appointments or answer time-sensitive queries, the LLM needs to know the current real-world state.

The Core API uses a utility (src/utils/promptVariables.ts) to dynamically inject temporal context into the prompt just before sending it to the LLM.

By default, it calculates time based on the Europe/Paris timezone (unless overridden by the agent’s configuration), ensuring that time-based tool calls (like scheduling a calendar event “tomorrow afternoon”) are accurately resolved.

The following variables are automatically parsed and replaced in any agent’s prompt:

Model Routing

The core conversational logic checks the agent’s llm configuration parameter. Based on this, it routes the formatted prompt, conversation history, and audio streams to the appropriate provider SDK:

• Google (@google/genai): Utilized for complex multimodal tasks, heavy tool calling, and high-speed inferences.
• OpenAI (openai): Available as an alternative engine for specific use cases or client preferences.