AI Agent: Local LLM Performance Analyzer & Benchmarking Tool

Unlock Optimized AI with this Local LLM Performance Analyzer

This AI Agent empowers you to systematically test, analyze, and compare the performance of various Large Language Models (LLMs) running locally via LM Studio. Stop guessing and start making data-driven decisions about which local LLM best suits your specific business automation needs. This agent automates the prompting, response gathering, and analysis, providing clear metrics for evaluation.

Key Features & Benefits

• Automated LLM Benchmarking: Connects directly to your LM Studio instance to retrieve a list of loaded models and iteratively tests each one based on your input prompt.
• Dynamic Prompting & Model Iteration: Sends your test prompts to multiple local LLMs fetched from LM Studio dynamically.
• AI-Driven Response Analysis: Utilizes a custom code node to calculate crucial performance and quality metrics for each LLM's response, including:
  • Flesch-Kincaid Readability Score
  • Word Count & Sentence Count
  • Average Sentence Length & Average Word Length
  • Response Time (latency calculation between start and end markers).
• Customizable System Prompts: Tailor system messages in the 'Add System Prompt' node to guide LLM behavior (e.g., for conciseness, specific tone, or reading level) ensuring fair comparisons across models.
• Adjustable LLM Parameters: Fine-tune settings like Temperature, Top P, and Presence Penalty directly within the 'Run Model with Dunamic Inputs' node for advanced model behavior control during tests.
• Comprehensive Reporting (Optional): Automatically logs detailed test results (prompt, model, response, all calculated metrics, timestamps) to a Google Sheet for easy review, comparison, and historical tracking.
• Streamlined Evaluation: Quickly identify the most performant, readable, or concise local LLM for your applications, saving significant manual effort in model selection.

• An n8n instance (Cloud or self-hosted).
• LM Studio installed on a local or network-accessible machine.
• Desired LLMs downloaded and loaded into the LM Studio server.
• The IP address and port of the running LM Studio server (the workflow defaults to port 1234 and expects an OpenAI-compatible API endpoint at /v1).
• An OpenAI credential configured in n8n. This can be a dummy credential (e.g., API key set to 'none') if your LM Studio server doesn't require authentication, as the Langchain OpenAI node expects this field.
• (Optional) Google Cloud Platform project with Google Sheets API enabled and OAuth2 credentials configured in n8n if you wish to log results to Google Sheets.

1. Download the n8n workflow JSON file.
2. Import the workflow into your n8n instance.
3. LM Studio Configuration:
   • Ensure LM Studio is installed and running.
   • Load the LLMs you want to test into LM Studio.
   • Start the LM Studio server (usually found in the "Local Server" tab). Note down its IP address and port (e.g., http://localhost:1234 or http://YOUR_LM_STUDIO_IP:1234).
4. Update LM Studio URLs in n8n:
   • In the 'Get Models' (HTTP Request) node: Set the URL parameter to http://YOUR_LM_STUDIO_IP:PORT/v1/models (replace YOUR_LM_STUDIO_IP:PORT with your actual server address and port).
   • In the 'Run Model with Dunamic Inputs' (LM Chat OpenAI) node: Go to the 'Options' tab and set the Base URL to http://YOUR_LM_STUDIO_IP:PORT/v1.
5. Configure LLM Node Credentials: In the 'Run Model with Dunamic Inputs' node, select your configured OpenAI credential. If LM Studio doesn't need an API key, ensure your selected credential handles this (e.g., a dummy key like 'NA' or 'test').
6. (Optional) Customize System Prompt: Modify the text in the 'Add System Prompt' (Set) node (parameter system_prompt) to define a consistent instruction for all tested LLMs.
7. (Optional) Configure Google Sheets Logging:
   • Create a new Google Sheet.
   • Add headers in the first row: Prompt, Time Sent, Time Received, Total Time Spent, Model, Response, Readability Score, Average Word Length, Word Count, Sentence Count, Average Sentence Length.
   • In the 'Save Results to Google Sheets' node:
     • Select or create your Google Sheets OAuth2 credentials.
     • Enter your Google Sheet's Document ID (from its URL).
     • Select the correct Sheet Name (e.g., Sheet1 or gid=0).
     • Verify the column mapping in the 'Columns' parameter matches your sheet headers.
8. (Optional) Adjust LLM Parameters: Modify Temperature, Top P, Presence Penalty in the 'Run Model with Dunamic Inputs' node's 'Options' tab as needed for your tests.
9. Trigger the Workflow: The workflow starts with the 'When chat message received' trigger. Open the chat interface for this workflow in n8n (usually accessible from the workflow editor by clicking the play button on the trigger or using the chat panel) and send a message. This message will be used as the initial prompt for the LLMs.
10. Review the output of the 'Analyze LLM Response Metrics' node and, if configured, your Google Sheet for the comparison. Refer to the 'Sticky Note' in the workflow for guidance on interpreting readability scores.
11. Activate the workflow for regular use. For distinct test runs, clear previous chat history if the chat trigger behavior carries it over (as suggested in the workflow's 'Pro Tip' sticky note).