AI Data Stack — Strategy

Where AI Data Stack is heading over the next 12 months, grounded in product-axis evidence and verbatim demand from the last 90 days. The judgment column is the engine's read — operators verify and refine.

Product trajectories

Data Labeling & RLHF Preference Data Platform — human-in-the-loop labeling service 🔥 hot

Opportunity: Frontier labs need 'audit-ready' RLHF preference data and PhD-grade evaluation; Scale's Meta entanglement just opened the door for Surge/Labelbox to absorb Anthropic+OpenAI+Google demand.

Hottest re-shuffle in ai-data-stack: Scale exited via $14.3B Meta deal, Google walked, Surge ($1B bootstrapped) + Labelbox Alignerr are catching the fall-out. Heavy A1 post-training hiring confirms RLHF/eval is the durable money — labeling is no longer about bounding boxes, it's about expert preference judgments.

Vector Database — vector database 🔥 hot

Opportunity: Buyers asking basic 'how to choose / local self-host / cost' questions while incumbents (Pinecone $100M ARR, Mongo 1M+ indexes) commoditize; S3 Vectors and in-process libs (Zvec, LEANN) threaten standalone DB category.

Vector DB is the most crowded sub-stack in ai-data-stack: 4 pure-plays (pinecone/weaviate/qdrant/chroma) all hiring AI/ML eng, but A2 chatter is shifting from 'what is it' to 'is the hype over / will S3 Vectors kill it'. Differentiation is collapsing onto price (Chroma Cloud, Pinecone Serverless) and local/embedded form factors.

RAG Infrastructure — retrieval pipeline ↗ rising

Opportunity: Production RAG is shifting from 'novel demo' to 'expensive ongoing infra' — community is openly debating cost ($2,400/mo), antipattern risk, and data-leakage of frameworks. Real spend exists, but no consensus on the reference stack.

RAG is the connective tissue of ai-data-stack but is in a credibility trough — heavy 'is this an antipattern' chatter alongside complaints about cost and leaky local modes. LlamaIndex is the only pure-play hiring visibly; Unstructured wins by being the picks-and-shovels dependency. Gap: opinionated, cost-tuned managed RAG.

Unstructured Document Parsing API — document parsing / ETL for LLMs → steady

Opportunity: Parsing 'hostile' / inconsistent enterprise documents (PDFs, Excel mapping specs, industrial protocols) into LLM-ready chunks remains a genuinely hard, high-volume problem.

Unstructured is the quiet kingmaker of ai-data-stack — embedded in LangChain/LlamaIndex, NVIDIA-backed, processing 1B docs/month. No real direct competitor surfaces in the evidence; risk is being commoditized by foundation-model native parsers.

LlamaIndex Workflows (Agent Orchestration) — agent workflow framework → steady

Opportunity: Developers questioning whether RAG is the right primitive for agents — opening for workflow/graph orchestration to be the new reference abstraction.

LlamaIndex is pivoting from index/RAG library to agent orchestration with Workflows, directly challenging LangGraph. Hiring spans applied AI + founding talent — actively scaling. Competition with LangChain ecosystem is the central risk.

Synthetic Data Generation — synthetic dataset platform → steady

Opportunity: Hyperscalers (NVIDIA, Databricks) are absorbing synthetic-data startups into their stacks, but the community is still asking 'is it practical?' — gap between platform conviction and developer trust.

Synthetic data is consolidating into platform features (NeMo Data Designer, Lakehouse) rather than standalone products. Pure-play opportunity narrowing; vertical/regulated-domain synthetic data may be the remaining wedge.

What the market is asking (last 90d)

• Which vector database do we like for local/selfhosted?
• Show HN: YourMemory, agentic memory is a pruning problem, not a hoarding problem
• Will Amazon S3 Vectors kill vector databases or save them?
• Vector database that can index 1B vectors in 48M
• The Vector Database Hype is Over (and That's Good)
• Why Your RAG Costs $2,400/Month (and How We Cut It by 73%)

See the Products and Hiring modules for the full landscape and who's investing in which direction.