Introduction
The technology industry evolves at an incredible pace. New frameworks emerge, AI technologies become mainstream, cloud platforms continue to expand, and companies constantly adjust their hiring priorities.
For professionals trying to understand the market, job portals provide access to thousands of job postings but very little insight into what those postings actually reveal about industry trends.
Questions such as:
- Which skills are becoming more valuable?
- What AI technologies are companies hiring for?
- Which job roles are growing fastest?
- What experience levels are currently in demand?
- Which locations have the highest hiring activity?
are difficult to answer manually because they require analyzing hundreds or thousands of job descriptions.
To solve this problem, I designed and developed a Real-Time AI Market Intelligence Platform using FastAPI, PostgreSQL, pgvector, LangGraph, Groq, Celery, Streamlit, n8n, and LangFuse.
The platform continuously collects job postings, extracts meaningful information, identifies hiring trends, supports AI-powered question answering, and automatically delivers market intelligence reports.
Rather than functioning as a traditional job portal, the system transforms raw job data into actionable market intelligence.
The Problem with Traditional Job Platforms
Most job platforms focus primarily on helping users find jobs.
While this solves the job discovery problem, it does not solve the market intelligence problem.
For example, if someone wants to know:
- What AI skills are companies hiring for?
- Which technologies are trending?
- What salary ranges are common?
- Which cities have the highest hiring demand?
- What experience levels are most frequently requested?
they would need to manually review hundreds of job postings and identify patterns themselves.
This process is time-consuming, inefficient, and nearly impossible to maintain as the market changes daily.
The goal of this project was to automate this entire process and provide real-time intelligence about the technology job market.
The Vision Behind the Project
Initially, the idea was simple.
Create a system that collects jobs from multiple sources and stores them in a database.
However, after collecting the first few hundred job postings, a much bigger opportunity became clear.
Every job posting contains valuable business signals.
A single job description can reveal:
- Skills being demanded
- Technologies being adopted
- Experience requirements
- Remote work trends
- Salary expectations
- Industry hiring priorities
If thousands of these signals are analyzed together, they create a powerful picture of the overall job market.
This realization transformed the project from a simple job aggregation platform into a Market Intelligence Platform.
System Architecture
The platform was designed as a layered architecture where each component focuses on a specific responsibility.
Technology Stack
- FastAPI for API development
- PostgreSQL for structured data storage
- pgvector for semantic vector search
- Sentence Transformers for embedding generation
- Groq LLM for AI-powered question answering
- LangGraph for intelligent query routing
- Celery and Redis for background processing
- Celery Beat for scheduling
- Streamlit for analytics dashboards
- n8n for workflow automation
- SMTP for report delivery
- LangFuse for AI observability
- Docker for containerization
The complete architecture looks like this:
Job Sources
↓
Scrapers
↓
PostgreSQL
↓
Embeddings
↓
pgvector
↓
Retriever
↓
LangGraph
↓
Groq
↓
FastAPI
↓
Streamlit Dashboard
Automation Layer:
Celery Beat
↓
Market Report
↓
n8n
↓
SMTP
↓
Email Delivery
Collecting Job Market Data
The first challenge was data collection.
Without data there can be no analytics, no trends, and no AI insights.
The platform continuously collects jobs from multiple job feeds and technology hiring sources.
Every few hours, Celery Beat automatically triggers job collection tasks.
The workflow is:
- Fetch jobs from external sources
- Normalize job information
- Validate and clean data
- Store jobs in PostgreSQL
- Generate embeddings
- Update analytics data
This ensures the platform always works with recent market information.
Designing the Database Layer
PostgreSQL was selected as the primary database because it provides reliability, indexing capabilities, and excellent analytical query performance.
Each job record stores:
- Job Title
- Description
- Source
- Skills
- Experience Level
- Salary Information
- Location
- Embedding Vector
- Collection Timestamp
This structured design enables both traditional analytics and AI-powered retrieval.
Transforming Raw Job Descriptions into Structured Data
Job descriptions are designed for humans, not machines.
Consider this example:
"Looking for a Python developer with AWS experience and 3+ years of software engineering experience."
A human instantly identifies:
- Python
- AWS
- 3+ Years Experience
Computers require explicit extraction logic.
To solve this challenge, dedicated extraction services were developed.
The platform automatically extracts:
- Technical Skills
- Experience Levels
- Salary Ranges
- Locations
- Remote Work Indicators
This structured information becomes the foundation for trend analysis.
Building Semantic Search with pgvector
Traditional search relies on exact keyword matching.
This creates limitations.
For example:
A user searches for:
AI Engineer
but the job description contains:
Machine Learning Engineer
Keyword search may fail even though the concepts are closely related.
To solve this problem, the platform uses embeddings and pgvector.
Every job description is converted into a 384-dimensional vector representation using Sentence Transformers.
The workflow becomes:
Job Description
↓
Embedding Model
↓
Vector Representation
↓
pgvector Storage
↓
Similarity Search
This allows the platform to understand meaning rather than simply matching words.
Building the AI Question Answering System
Once semantic search was working, the next goal was to allow users to ask questions directly.
Examples:
- What AI skills are companies hiring for?
- What are the responsibilities of an AI Solutions Manager?
- Which cloud technologies are trending?
- What jobs are currently in demand?
To achieve this, a Retrieval-Augmented Generation (RAG) architecture was implemented.
The workflow is:
User Question
↓
Retriever
↓
Relevant Jobs
↓
Context Creation
↓
Groq LLM
↓
Final Answer
The model does not generate answers blindly.
Instead, it first retrieves relevant jobs from the database and then generates an answer using actual market data.
Introducing LangGraph for Intelligent Routing
During development, an important observation emerged.
Not every question requires an LLM.
For example:
"What are the top skills this month?"
The answer already exists in the analytics database.
Calling an LLM would increase cost and latency unnecessarily.
To solve this problem, LangGraph was introduced.
LangGraph acts as a decision engine.
It determines what type of question the user is asking and routes it appropriately.
Possible routes include:
- Skill Trends
- Job Trends
- Salary Trends
- Experience Trends
- Location Trends
- RAG-based AI Answers
This significantly improved performance while reducing unnecessary AI calls.
Building Analytics and Market Insights
The platform generates multiple forms of analytics directly from collected job data.
Examples include:
- Top Skills
- Trending Technologies
- Most Demanded Roles
- Experience Distribution
- Location-Based Hiring Trends
- Salary Insights
For example, the system can identify trends such as:
- Python appearing in 58 job postings
- AWS appearing in 43 job postings
- LLM-related skills appearing in 17 job postings
This transforms individual job listings into meaningful market intelligence.
Creating the Streamlit Dashboard
Raw analytics are difficult to consume.
To make insights accessible, a Streamlit dashboard was developed.
The dashboard provides:
- Market Overview
- Trending Skills
- Salary Analysis
- Experience Analysis
- Job Explorer
- AI Chat Assistant
This enables users to explore market intelligence visually rather than querying raw data.
Automation with Celery and n8n
A major goal of the project was automation.
Users should not have to manually check the dashboard every day.
The platform automatically generates market reports.
The automation workflow is:
Celery Beat
↓
Collect Jobs
↓
Generate Report
↓
n8n Webhook
↓
SMTP
↓
Email Delivery
Reports are generated automatically every six hours and delivered through email workflows.
This transforms the platform from a passive dashboard into a proactive intelligence system.
Observability with LangFuse
AI systems can be difficult to debug.
If an answer is incorrect, understanding why becomes challenging.
LangFuse was integrated to provide observability.
It tracks:
- User Questions
- Retrieved Documents
- Prompts
- Model Responses
- Latency
- Token Usage
- Routing Decisions
This makes it easier to monitor and improve AI performance over time.
Engineering Challenges Faced During Development
Database Schema Mismatches
As the project evolved, new fields such as location and embeddings were introduced.
This occasionally created mismatches between SQLAlchemy models and PostgreSQL tables.
Schema synchronization and migrations solved the issue.
pgvector Installation Issues
PostgreSQL initially lacked pgvector support.
The extension had to be installed and configured before vector search could be enabled.
LangGraph Routing Problems
Some trend-related questions were incorrectly routed to the RAG system.
The routing logic was refined to improve intent detection accuracy.
Celery Task Registration Errors
Task discovery issues initially prevented workers from executing scheduled jobs.
Proper task registration and import management resolved the problem.
Google OAuth Restrictions
Google OAuth verification limitations caused email delivery issues.
The system was redesigned to use SMTP-based email delivery through n8n, resulting in a more reliable solution.
Key Learnings
This project provided hands-on experience across multiple engineering domains.
- Data Engineering
- Backend Development
- Vector Databases
- Retrieval-Augmented Generation
- Workflow Automation
- AI Observability
- Analytics Systems
- Scalable Architecture Design
Perhaps the most important lesson was that successful AI products require much more than an LLM.
Data pipelines, retrieval quality, orchestration, monitoring, automation, and user experience are equally important components.
Conclusion
What started as a simple job aggregation idea evolved into a complete AI-powered Market Intelligence Platform.
The system continuously collects job data, analyzes trends, performs semantic search, answers natural language questions, delivers automated reports, and provides actionable insights into the technology job market.
By combining FastAPI, PostgreSQL, pgvector, Groq, LangGraph, Celery, Streamlit, n8n, and LangFuse, the platform demonstrates how modern AI engineering, data engineering, and automation technologies can work together to transform raw information into business intelligence.
The result is not simply a dashboard or chatbot. It is an end-to-end intelligence system designed to help users understand a rapidly changing job market through data, analytics, and AI.