ZFushou is a large scale user message processing system built to automatically organize, track, and analyze thousands of incoming messages across community channels.

The core problem was that discussions rarely happen in clean, isolated threads. A user might report an API issue, another user asks a billing question a few messages later, and someone else continues the original API discussion. Over time, related messages become scattered across channels and timelines, making it difficult to understand what people are discussing, identify recurring problems, or retrieve historical context.

To solve this, I built a real time semantic routing engine combined with a large scale batch analysis pipeline.

The live engine runs continuously and processes new messages every 15 seconds. Each message is cleaned, embedded, and compared against active discussion states stored in a vector database. Rather than relying on channels, threads, keywords, or manual tagging, the system determines whether a message belongs to an existing discussion by measuring semantic similarity between the new message and the running centroid of active discussions.

For every incoming message, the engine generates an embedding using Cloudflare Workers AI and searches Qdrant for the nearest active discussion. Different similarity thresholds are applied depending on whether the message originates from the same channel or a different one. If a suitable match is found, the message is attached to that discussion. If no active match exists, the system attempts to reopen recently closed discussions before creating a completely new discussion state.

A key challenge was detecting when a discussion naturally shifts into a different topic. To solve this, I implemented a semantic boundary detection algorithm based on centroid cohesion tracking. Every active discussion maintains a continuously updated centroid vector representing the current topic. When a new message arrives, the system measures both its similarity to the active centroid and the drop in cohesion compared to previous messages. A significant drop combined with low similarity signals that the conversation has moved into a different subject. The previous discussion is automatically closed and a new one begins without requiring any manual intervention.

The centroid itself is updated incrementally after every message using a running average across embeddings, allowing the discussion representation to evolve naturally as more context is added while remaining computationally efficient.

The engine also maintains lifecycle management for active discussions. Discussions can be closed through semantic topic shifts, inactivity timeouts, explicit resolution detection, or scheduled resets. Previously closed discussions can later be reopened if new messages are semantically similar to the original topic. When this happens, the system restores historical context, rebuilds discussion state from stored vectors, and reanalyzes the entire discussion history.

To keep discussion summaries and status information up to date, I implemented event driven analysis triggers. Rather than sending every message through an LLM, the system selectively triggers analysis based on message volume, severity indicators, resolution signals, and discussion progression. This significantly reduces inference cost while maintaining current state awareness.

The live system is designed for production reliability. Processing is coordinated through Redis distributed locking to prevent duplicate workers from operating simultaneously. Lock ownership is continuously refreshed during execution, and processing stops immediately if ownership is lost. Batch orchestration includes fault tolerant retries with exponential backoff and automatic recovery from transient failures.

Alongside the real time engine, I built a large scale batch analysis pipeline that runs every 12 hours across historical data.

Messages are loaded from Supabase in paginated batches and cleaned to remove bots, links, and low value noise. Every message is embedded individually and passed through a custom TextTiling implementation adapted specifically for chat environments.

The segmentation algorithm uses a sliding window approach. Three messages on the left side of a boundary candidate are compared against three messages on the right using cosine similarity. When similarity drops significantly, the algorithm identifies a potential topic transition. Depth scores are calculated to measure the strength of each transition, smoothed using moving averages, and filtered through thresholding logic to distinguish genuine topic changes from normal conversational variation.

After segmentation, discussion blocks are reconstructed with contextual history. Each message is paired with its previous two messages while ensuring context never crosses detected topic boundaries. This preserves conversational meaning while preventing unrelated discussions from being merged together.

These contextual blocks undergo a second embedding pass. The first embedding pass is optimized for topic boundary detection, while the second is optimized for retrieval quality and semantic search. This two stage architecture produced significantly better search performance than relying on single message embeddings alone.

To categorize discussions, I implemented a two stage classification workflow. Instead of defining categories manually, the system first samples representative discussion segments and allows an LLM to discover recurring themes directly from the data. Categories identified during this phase are then used to classify the entire dataset. This keeps classification grounded in actual user behavior rather than assumptions made during development.

All semantic vectors are stored in Qdrant with metadata such as timestamps, channels, similarity scores, and boundary metrics. Structured analytics data is stored in Supabase for reporting and aggregation.

The final system transforms thousands of unstructured messages into searchable, organized discussion groups. Teams can track recurring issues, identify emerging topics, analyze trends over time, retrieve historical discussions through semantic search, and understand how user concerns evolve without manually reading large volumes of messages.

Technically, the project combines distributed systems, vector search, semantic segmentation, retrieval architecture, real time stream processing, LLM driven analysis, state management, and production backend engineering into a single end to end platform.