AI Agent: Crop Image Vectorizer & Qdrant Uploader

Unlock Advanced Image Analysis with this AI Agent

This n8n workflow acts as a powerful AI Agent designed to process image datasets for sophisticated machine learning applications. It automates the pipeline of fetching images (e.g., agricultural crops) from Google Cloud Storage, generating meaningful vector embeddings using Voyage AI's multimodal capabilities, and indexing them into a Qdrant vector database. This prepares your data for tasks like visual similarity search, K-Nearest Neighbors (KNN) classification, and anomaly detection.

This agent is the foundational first step (e.g., 1 of 3 for anomaly detection, 1 of 2 for KNN classification) in building more complex AI systems. It handles batch processing, Qdrant collection setup (including payload indexing on metadata like crop_name), and even allows for specific data filtering to tailor datasets for particular AI model training or testing scenarios (like excluding 'tomato' images to later test anomaly detection capabilities).

Key Features & Benefits

• Automated Image Ingestion: Fetches image datasets directly from Google Cloud Storage based on bucket and prefix.
• AI-Powered Embeddings: Utilizes Voyage AI's multimodal models to create rich vector representations of your images, capturing complex visual features.
• Efficient Batch Processing: Processes images, generates embeddings, and uploads to Qdrant in configurable batches, optimizing for large datasets and API rate limits.
• Qdrant Vector Database Integration: Seamlessly creates or updates Qdrant collections, uploads image vectors along with metadata (e.g., crop_name, image URL), and sets up payload indexes for efficient querying.
• Customizable Data Preparation: Includes steps to extract metadata from image paths and filter datasets for specific use cases (e.g., preparing a dataset for anomaly detection by excluding certain classes during upload).
• Scalable Foundation: Provides the essential groundwork for building AI-driven visual search, image classification, and anomaly detection systems for your business.

• An n8n instance (Cloud or self-hosted).
• Qdrant Cloud cluster URL and API Key (Free Tier available is sufficient).
• Voyage AI API Key (for multimodal embeddings).
• Google Cloud Storage bucket containing your image dataset, and OAuth2 credentials for access.
• Your image dataset should consist of images accessible via public URLs; this workflow constructs these from Google Cloud Storage links.

1. Download the n8n workflow JSON file.
2. Import the workflow into your n8n instance.
3. Configure Qdrant Variables: In the 'Qdrant cluster variables' node, update qdrantCloudURL with your cluster's URL. Adjust collectionName as needed. The VoyageEmbeddingsDim (e.g., 1024) should match your Voyage AI model; batchSize (e.g., 4) can be tuned based on performance and API limits.
4. Set Up Qdrant Credentials: For nodes 'Check Qdrant Collection Existence', 'Create Qdrant Collection', 'Payload index on crop_name', and 'Batch Upload to Qdrant', select or create new Qdrant API credentials (typically an API key).
5. Configure Google Cloud Storage: In the 'Google Cloud Storage' node, select/create your OAuth2 credentials. Set the bucketName to your image bucket and listFilters.prefix to the folder path of your dataset (e.g., agricultural-crops).
6. Set Up Voyage AI Credentials: In the 'Embed crop image' (HTTP Request) node, select or create new HTTP Header Auth credentials. The header name should be Authorization and the value Bearer YOUR_VOYAGE_AI_API_KEY.
7. Verify Field Extraction: Check the 'Get fields for Qdrant' (Set) node. Ensure it correctly extracts publicLink for images and metadata like cropName from your GCS object paths. The current setup assumes paths like .../folder_as_crop_name/image.jpg.
8. Adjust Filtering (Optional): The 'Filtering out tomato to test anomalies' (Filter) node is pre-configured to exclude images with cropName equal to 'tomato'. Modify or remove this if not needed for your specific use case.
9. Review Batching Logic: The 'Split in batches, generate uuids for Qdrant points' (Code) node uses Python to batch items and generate UUIDs. Ensure batchSize from variables is used correctly.
10. Activate the workflow to begin vectorizing and uploading your image dataset to Qdrant.