AI Web Content Fetcher & Processor Agent (ReAct)

Unlock Intelligent Web Content Extraction with this AI Agent

This n8n workflow implements a ReAct (Reasoning and Acting) AI Agent designed for sophisticated web content retrieval and processing. Triggered by a manual chat message, the agent leverages OpenAI's GPT-4 model to understand user requests. It then utilizes a custom 'HTTP_Request_Tool' to fetch web pages. The agent's strength lies in its ability to not just fetch, but also to meticulously clean and prepare the HTML content. It extracts the main body, strips unnecessary scripts, styles, and tags, optionally simplifies content by removing links and images, and finally converts the cleaned HTML to Markdown. This process ensures that the data fed back to the AI or subsequent automation steps is concise, relevant, and in a machine-readable format, while also handling potential errors like invalid URLs or oversized pages gracefully.

Key Features & Benefits

• ReAct AI Agent Framework: Employs a LangChain ReAct agent for robust reasoning and tool utilization, driven by OpenAI's GPT-4.
• Intelligent Web Fetching: The agent uses a custom tool to perform HTTP requests based on AI-interpreted instructions.
• Advanced Content Cleaning: Automatically isolates the HTML body and removes scripts, styles, iframes, comments, and other non-essential elements.
• Content Simplification Option: Can further reduce content size by stripping all URLs and image sources, focusing on textual information for efficient LLM processing.
• Markdown Conversion: Transforms processed HTML into clean Markdown, ideal for AI consumption or structured data use.
• Dynamic Parameterization: Supports query parameters (e.g., url, method=['full'|'simplified'], maxlimit) for flexible fetching control, guided by the AI's understanding.
• Error Handling & Feedback: Manages HTTP errors and content length limits, providing clear feedback to the agent or user.
• Chat-Driven Interaction: Easily initiated and controlled via manual chat inputs, allowing for flexible and dynamic tasking.

• An n8n instance (Cloud or self-hosted).
• OpenAI API Key with access to a suitable model (e.g., gpt-4-1106-preview or gpt-4-turbo).
• n8n LangChain & AI-related nodes enabled in your n8n instance if self-hosting (usually enabled by default on n8n Cloud).

1. Download the n8n workflow JSON file.
2. Import the workflow into your n8n instance.
3. Configure the 'OpenAI Chat Model' node: select your OpenAI credential (e.g., 'OpenAi account') and ensure the chosen model (e.g., gpt-4-1106-preview) is suitable for your needs and API key access.
4. Review the 'HTTP_Request_Tool' node: its 'Description' field is crucial as it instructs the ReAct AI Agent on how to use the tool and format the input string (e.g., "?url=VALIDURL&method=SELECTEDMETHOD"). No changes are typically needed here to start.
5. The workflow is triggered by the 'On new manual Chat Message' node. To test, run the workflow manually and enter a prompt for the agent in the chat input, for example: "Fetch the content of example.com using the simplified method and tell me what it's about."
6. Examine the execution flow in n8n, particularly the 'ReAct AI Agent' node's output, to see the agent's thoughts and actions, and the final output from the HTTP request processing part of the workflow.
7. Activate the workflow to use it interactively via the n8n chat interface.