AI Survey Insights Agent with OpenAI & Qdrant

Unlock Deep Survey Insights with this AI Agent

This n8n workflow transforms raw survey data into actionable intelligence. It connects to your Google Sheet, ingests responses, and then employs a sophisticated AI-driven process. First, it converts answers into vector embeddings using OpenAI for deep semantic understanding. These embeddings are stored and indexed in a Qdrant vector database. The agent then intelligently groups similar responses for each question using K-means clustering to uncover underlying themes and popular opinions. Finally, it leverages OpenAI's powerful language models to summarize these themed clusters, extract key insights, determine sentiment, and identify the participants associated with each group. The generated insights are then neatly exported back to a new sheet in your Google Spreadsheet.

This AI Agent empowers you to move beyond surface-level survey analysis, automating the extraction of rich, nuanced understanding from qualitative feedback at scale.

Key Features & Benefits

• Automated Data Ingestion: Seamlessly pulls survey responses directly from Google Sheets.
• AI-Powered Vectorization: Uses OpenAI's text-embedding-3-small (or configurable) model to convert text responses into meaningful vector embeddings.
• Efficient Vector Storage: Leverages Qdrant for storing and querying vectorized survey answers, enabling quick retrieval of semantically similar responses.
• Intelligent Theme Detection: Applies K-means clustering (via a Python script with scikit-learn) to vectorized answers, automatically grouping similar responses to identify dominant themes and opinions for each question.
• LLM-Driven Insight Generation: Utilizes an OpenAI chat model (e.g., gpt-4o-mini) through Langchain's Information Extractor to summarize clustered responses, provide actionable insights, and perform sentiment analysis on each theme.
• Detailed Reporting: Exports structured insights, including the question, summarized insight, sentiment, number of responses in the cluster, and participant identifiers, back to Google Sheets.
• Scalable Analysis: Efficiently processes large volumes of survey data, saving significant manual effort and time.
• Deeper Understanding: Uncovers subtle patterns and qualitative nuances often missed in manual analysis.

• An n8n instance (Cloud or self-hosted).
• OpenAI API Key with access to embedding models (e.g., text-embedding-3-small) and chat models (e.g., gpt-4o-mini).
• Google Sheets API credentials configured in n8n (OAuth2 recommended).
• A Qdrant instance (local, cloud, or Docker) accessible by your n8n instance. API key if Qdrant is secured.
• For the 'Apply K-means Clustering Algorithm' Code node: If your n8n environment doesn't automatically handle Python package installations for Code nodes, ensure numpy and scikit-learn are installed in the Python environment used by n8n. (Note: n8n typically attempts to install required packages; a slight delay may occur on the first run of this node).

1. Download the n8n workflow JSON file.
2. Import the workflow into your n8n instance.
3. Initial Data Ingestion Setup (Run this part first): a. Configure the 'Get Survey Results' (Google Sheets) node: Select your Google Sheet. Ensure the first row contains headers (questions) and subsequent rows contain participant responses. b. Configure the 'Embeddings OpenAI' node: Select your OpenAI API credentials. c. Configure the 'Qdrant Vector Store' node: Enter your Qdrant instance URL (e.g., http://qdrant:6333), API key (if applicable), and set the 'Qdrant Collection Name' (e.g., ux_survey_insights). d. Run the workflow from the 'When clicking ‘Test workflow’' node up to the 'Qdrant Vector Store' node to populate your Qdrant collection. This step needs to be done once per survey dataset, or if new data is added and needs vectorization.
4. Insight Generation Setup (Main analysis loop): a. The workflow uses an 'Execute Workflow Trigger' which means the second part of the flow (starting from 'Execute Workflow Trigger' node) runs as a sub-workflow. b. Configure the 'Get Sheet Details' (HTTP Request) node: Ensure it points to the Google Sheets API and uses your Google Sheets credentials. c. Configure 'Set Variables' node: The collectionName should match the one used in the 'Qdrant Vector Store' node. surveyName is derived from the Google Sheet title. insightsSheetName defines the output sheet name. d. Configure 'Get Survey Headers' (Google Sheets) node: This should point to the same Google Sheet as 'Get Survey Results'. e. Configure 'Find All Answers' & 'Get Payload of Points' (HTTP Request) nodes: Ensure the Qdrant URL and collection name are correctly templated or set to match your Qdrant setup. f. Configure the 'OpenAI Chat Model' node: Select your OpenAI API credentials. g. Review the 'Survey Insights Agent' (Information Extractor) node: The system prompt and JSON schema define how insights are extracted. Customize if necessary. h. Configure 'Export To Sheets' and 'Export To Sheets1' (Google Sheets) nodes: Ensure they point to the desired Google Sheet document for outputting insights.
5. Triggering the Analysis: a. After initial data ingestion (step 3d), you can trigger the full insight generation process via the 'Trigger Insights' (Execute Workflow) node. This will then run the sub-workflow part starting from 'Execute Workflow Trigger'. b. Alternatively, if you've set up the workflow to be triggered (e.g., manually via the 'Execute Workflow Trigger' or on a schedule), ensure the 'Prep Values For Trigger' node correctly passes the sheetID and sheetName from the initial 'Get Survey Results' node's parameters.
6. Activate the workflow. The main workflow path vectorizes and stores data, then calls itself (via Execute Workflow and Execute Workflow Trigger) to perform the per-question analysis and insight generation.