Back to Projects

Problem

Most AI research assistants and search engines focus on accuracy and similarity, which often narrows discovery.
However, creative breakthroughs frequently arise from serendipity — the unexpected yet meaningful connection between distant ideas.
The challenge was to design an AI agent that intentionally injects “useful surprise” into the research process, helping users make lateral, cross-domain discoveries instead of repetitive, same-domain suggestions.

Solution Overview

The Serendipity Engine is a prototype AI agent that suggests serendipity cards — short, contextually relevant but novel ideas derived from diverse sources like arXiv and Wikipedia.
It leverages large language model embeddings to balance relevance (semantic similarity) and novelty (embedding distance) and visualizes the relationships through an interactive concept map.

System flow:
User Input → Context Scraper → Cross-Domain Linker → Inspiration Generator → Visualization Layer → Feedback & Memory

Each stage refines the creative discovery loop:

  • Extract context from the user’s research text.
  • Retrieve conceptually related but diverse snippets across domains.
  • Score them on semantic relevance and conceptual novelty.
  • Package the best ones as “serendipity cards” with rationales and prompts.
  • Visualize relationships and adapt over time via user feedback.

Architecture & Design

Backend

Built with FastAPI, the backend exposes modular endpoints:

  • /context/parse – Extracts key topics, entities, and key phrases using spaCy.
  • /context/index – Embeds the user’s context and external corpus (arXiv, Wikipedia) into ChromaDB.
  • /cards/generate – Computes a centroid embedding, retrieves candidate snippets, filters by thresholds, and generates serendipity cards.
  • /feedback and /memory – Record user preferences and adapt novelty thresholds.
  • /metrics and /survey – Collect performance metrics and user feedback for evaluation.

Data is persisted in SQLite (for metadata, preferences, and logs) and ChromaDB (for vector search).

Frontend

The UI, built in Streamlit, provides:

  • Text input and context indexing controls.
  • A responsive grid of serendipity cards with sources and rationales.
  • Interactive Plotly concept map showing semantic relationships.
  • Feedback buttons (👍 / 👎 / 💾) and a novelty preference slider.
  • Metrics and survey dashboards for evaluation.

Key Data Flow

1. User provides text
2. Context Scraper → spaCy entities & topics
3. Embedding Pipeline → SentenceTransformer (MiniLM-L6-v2)
4. Cross-Domain Linker → cosine similarity & distance filtering
5. Inspiration Generator → heuristic rationale + actionable prompt
6. Visualization Layer → Streamlit + Plotly
7. Memory Module → stores feedback to tune novelty window

Technical Highlights

Controlled Novelty Algorithm

Balanced semantic relevance (cosine similarity ≥ 0.7) and novelty (cosine distance 0.6–0.8) to ensure insights were surprising but not random.

Dynamic Backoff Filtering

Graceful fallback mechanism widens thresholds if too few matches pass strict filters — preventing empty responses.

Adaptive Personalization

A memory module tunes the novelty window based on the user’s saved preference (0 = familiar, 1 = novel).

Semantic Indexing Pipeline

Sentence-transformer embeddings + ChromaDB vector store for fast cross-domain retrieval.

Instrumentation

Built-in /metrics endpoint logs latency, relevance, and novelty averages, with p50/p95 performance tracking.

Full-Stack Integration

FastAPI backend + Streamlit frontend communicate via REST with CORS enabled for local or remote deployment.

My Contributions

I co-designed and implemented the entire prototype over multiple structured phases:

  • Backend architecture: FastAPI services, SQLite schema, ChromaDB integration.
  • Context Scraper & Embedding pipeline: spaCy-based entity extraction and topics extraction, sentence-transformer encoding, caching.
  • Cross-Domain Linker: centroid-based retrieval and dual-threshold filtering for relevance and novelty.
  • Heuristic Card Generator: rationales and actionable prompts without external LLM calls.
  • Streamlit Interface: interactive card grid, concept map visualization, and feedback module.
  • Evaluation & Metrics: implemented runtime profiling, survey collection, and performance summary endpoints.

Results & Impact

  • Generated meaningful cross-domain insights linking topics like urban logistics to biological optimization and cargo-bike mobility.
  • Achieved avg. semantic relevance ≈ 0.72, avg. novelty ≈ 0.67, and p95 latency < 2 s for card generation.
  • Built a fully functioning end-to-end research assistant prototype usable for creativity and ideation tasks.
  • Established a reusable codebase and evaluation framework for future LLM creativity experiments.

What I Learned

  • How to balance semantic similarity and diversity in vector search to simulate “useful surprise.”
  • Practical experience combining LLMs, embeddings, and UX visualization into a cohesive system.
  • The importance of feedback loops and evaluation metrics in creative AI tools.
  • Hands-on exposure to production-style FastAPI architecture, data caching, and Streamlit UI integration.
  • Broader insight into designing intentional serendipity as a measurable, repeatable process.