Merge pull request #36 from Shishir435/feature/file-upload-processing

Feature/file upload processing

Author: Shishir435

README.md

@@ -11,16 +11,11 @@
   <img src="https://img.shields.io/badge/Brave-Supported-orange?logo=brave&style=for-the-badge" alt="Brave Supported" />
   <img src="https://img.shields.io/badge/Edge-Supported-blue?logo=microsoftedge&style=for-the-badge" alt="Edge Supported" />
   <img src="https://img.shields.io/badge/Opera-Supported-red?logo=opera&style=for-the-badge" alt="Opera Supported" />
-  <img src="https://img.shields.io/badge/Firefox-Experimental-lightgrey?logo=firefox-browser&style=for-the-badge" alt="Firefox Experimental" />
+  <img src="https://img.shields.io/badge/Firefox-Supported-lightgrey?logo=firefox-browser&style=for-the-badge" alt="Firefox Supported" />
 </div>
 
 ---
 
-> ✅ Works with any Chromium-based browser: **Chrome**, **Brave**, **Edge**, **Opera**, and **Chromium**  
-> 🦊 **Firefox support** available via [temporary addon installation](https://extensionworkshop.com/documentation/develop/temporary-installation-in-firefox/) (manual permissions setup required).
-
----
-
 ## 🚀 Get Started — Install Now
 
 <div align="center">
@@ -68,6 +63,22 @@
 - 📋 **Copy & Regenerate** – Quickly rerun or copy AI responses
 - ⚡ **Streaming Responses** – Real-time streaming with typing indicators
 
+### 🧠 Embeddings & Semantic Search (Beta v0.3.0)
+- 🔍 **Semantic Chat Search** – Search chat history by meaning, not just keywords
+- 📊 **Vector Database** – IndexedDB-based vector storage with optimized cosine similarity
+- 🎯 **Smart Chunking** – 3 strategies: fixed, semantic, hybrid (configurable)
+- 🚀 **Optimized Search** – Pre-normalized vectors, caching, early termination
+- 🔧 **Configurable** – Chunk size, overlap, similarity threshold, search limits
+- 📁 **Context-Aware** – Search across all chats or within current session
+
+### 📎 File Upload & Processing (Beta v0.3.0)
+- 📄 **Text Files** – Support for .txt, .md and text based files
+- 📕 **PDF Support** – Full text extraction from PDF documents
+- 📘 **DOCX Support** – Extract text from Word documents
+- ⚙️ **Auto-Embedding** – Automatic embedding generation for uploaded files
+- 📊 **Progress Tracking** – Real-time progress indicators during processing
+- 🎛️ **Configurable Limits** – User-defined max file size in settings
+
 
 ### 🌐 Webpage Integration
 - 🧠 **Enhanced Content Extraction** – Advanced extraction with multiple scroll strategies (none, instant, gradual, smart)
@@ -83,7 +94,7 @@
 - 🌓 **Dark Mode** – Beautiful dark theme with smooth transitions
 - 📝 **Prompt Templates** – Create, manage, and use custom prompt templates (Ctrl+/)
 - 🔊 **Advanced Text-to-Speech** – Searchable voice selector with adjustable speech rate & pitch
-- 🎚️ **Cross-Browser Compatibility** – Works with Chrome, Brave, Edge, Opera, Vivaldi, LibreWolf, and more
+- 🎚️ **Cross-Browser Compatibility** – Works with Chrome, Brave, Edge, Opera
 - 🧪 **Voice Testing** – Test voices before using them
 
 ### 🔒 Privacy & Performance
@@ -98,29 +109,36 @@
 ## 🧩 Tech Stack
 
 ### Frontend
-- **TypeScript** – Type-safe development
-- **React 18** – Modern UI framework
-- **Vite** – Fast build tool
-- **Plasmo** – Chrome extension framework
-- **Shadcn UI** – Professional component library (Radix UI primitives)
-- **Tailwind CSS** – Utility-first styling
-- **Zustand** – Lightweight state management
-- **Dexie** – IndexedDB wrapper for chat storage
-
-### Backend & APIs
-- **Ollama** – Local LLM backend
-- **Chrome Extension APIs** – `declarativeNetRequest`, `storage`, `sidePanel`, `tabs`
+- **[TypeScript](https://www.typescriptlang.org/)** – Type‑safe development
+- **[React 18](https://reactjs.org/)** – Modern UI framework
+- **[Plasmo](https://docs.plasmo.com/)** – Chrome‑extension framework
+- **[Shadcn UI](https://ui.shadcn.com/)** – Professional component library (Radix UI primitives)
+- **[Radix UI](https://www.radix-ui.com/)** – Accessible UI primitives 
+- **[Tailwind CSS](https://tailwindcss.com/)** – Utility‑first styling
+- **[Lucide React](https://lucide.dev/)** – Icon library
+- **[Zustand](https://github.com/pmndrs/zustand)** – Lightweight state management
+- **[Dexie](https://dexie.org/)** – IndexedDB wrapper for chat storage
+
+- **[webextension‑polyfill](https://github.com/mozilla/webextension-polyfill)** – Promise‑based browser extension API wrapper
+
+### Backend & APIs
+- **[Ollama](https://ollama.com/)** – Local LLM backend
+- **[Chrome Extension APIs](https://developer.chrome.com/docs/extensions/)** – `declarativeNetRequest`, `storage`, `sidePanel`, `tabs`
 
 ### Content Processing
-- **Defuddle** – Advanced content extraction
-- **Mozilla Readability** – Content extraction fallback
-- **highlight.js** – Code syntax highlighting
-- **markdown-it** – Markdown rendering
+- **[Defuddle](https://github.com/kepano/defuddle)** – Advanced content extraction
+- **[Mozilla Readability](https://github.com/mozilla/readability)** – Content extraction fallback
+- **[highlight.js](https://highlightjs.org/)** – Code syntax highlighting
+- **[markdown-it](https://github.com/markdown-it/markdown-it)** – Markdown rendering
+- **[pdfjs‑dist](https://github.com/mozilla/pdfjs-dist)** – PDF parsing and rendering
+- **[dompurify](https://github.com/cure53/DOMPurify)** – HTML sanitization
+- **[html2pdf.js](https://github.com/eKoopmans/html2pdf.js)** – Convert HTML to PDF
+- **[mammoth](https://github.com/mwilliamson/mammoth.js)** – DOCX to HTML conversion
 
 ### Developer Tools
-- **Biome** – Fast formatter & linter
-- **TypeScript** – Strict type checking
-- **Husky** – Git hooks
+- **[Biome](https://biomejs.dev/)** – Fast formatter & linter
+- **[TypeScript](https://www.typescriptlang.org/)** – Strict type checking
+- **[Husky](https://typicode.github.io/husky/)** – Git hooks
 
 ---
 
@@ -304,13 +322,24 @@ Load as a [temporary extension](https://extensionworkshop.com/documentation/deve
 
 ### 📁 Code Structure
 
-- `src/`: Core logic and components
-- `background.ts`: API bridge + streaming
-- `sidepanel.tsx`: Main chat UI
-- `options.tsx`: Settings page
-- `content.ts`: Summarizer / Readability
-- `lib/`: Utility functions
-- `hooks/`, `features/`, `context/`: Modular structure for maintainability
+```
+src/
+├── background/        # Background service worker & API handlers
+├── sidepanel/         # Main chat UI
+├── options/           # Settings page
+├── features/          # Feature modules
+│   ├── chat/          # Chat components, hooks, semantic search
+│   ├── model/         # Model management & settings
+│   ├── sessions/      # Chat session management
+│   ├── prompt/        # Prompt templates
+│   └── tabs/          # Browser tab integration
+├── lib/               # Shared utilities
+│   └── embeddings/    # Vector embeddings & semantic search
+├── components/        # Shared UI components (Shadcn)
+└── hooks/             # Shared React hooks
+```
+
+**Architecture**: Feature-based organization with separation of concerns (components, hooks, stores). Zustand for global state, React hooks for local state.
 
 ---
 
@@ -334,7 +363,7 @@ Explore: [Ollama Model Library](https://ollama.com/library)
 
 ---
 
-## 🧪 Firefox Support (Experimental)
+## 🧪 Firefox Support
 
 Ollama Client is a Chrome Manifest V3 extension. To use in Firefox:
 
@@ -368,12 +397,32 @@ Here’s what’s coming up next in **Ollama Client**—grouped by priority:
 
 ### Embeddings & Semantic Search
 
-- [ ] Implement **Ollama Embedding Models**:
-  - [ ] Integration with Ollama embedding models (e.g., `nomic-embed-text`, `all-minilm`)
-  - [ ] Generate embeddings for chat messages and store in IndexedDB
-  - [ ] Semantic search over chat history
-  - [ ] Semantic filtering for chat sessions
-- [ ] Enable **Local RAG** over chats, PDFs, and image text
+- [x] Implement **Ollama Embedding Models**:
+  - [x] Integration with Ollama embedding models (e.g., `nomic-embed-text`, `mxbai-embed-large`)
+  - [x] Generate embeddings for chat messages and store in IndexedDB
+  - [x] Semantic search over chat history (global and per-chat)
+  - [x] Auto-embedding toggle and backfill functionality
+- [x] **Vector Search Optimization** (Phase 1 - Completed):
+  - [x] Brute-force cosine similarity with optimized computation
+  - [x] Pre-normalize embeddings on storage
+  - [x] Use Float32Array for better memory locality
+  - [x] Implement early termination for low similarity scores
+  - [x] Add search result caching (configurable TTL & max size)
+  - [x] Non-blocking computation (async chunking with yields)
+- [x] **Semantic Chat Search UI** (Beta v0.3.0 - Completed):
+  - [x] Search dialog with debounced input
+  - [x] Search scope toggle (all chats / current chat)
+  - [x] Grouped results by session
+  - [x] Similarity scores with % match display
+  - [x] Click to navigate and highlight message
+  - [x] Real-time loading indicators
+- [ ] **Advanced Vector Search** (Phase 2 - Future):
+  - [ ] WASM-based HNSW indexing via `hnswlib-wasm`
+  - [ ] Implement when dataset exceeds 10K vectors
+  - [ ] Expected performance gain: 10-100x faster searches
+  - [ ] Bundle size impact: ~200 KB (gzipped: ~100-200 KB)
+  - [ ] See `VECTOR_SEARCH_LIBRARIES_REPORT.md` for detailed analysis
+- [ ] Enable **Local RAG** over chats, PDFs, and uploaded files
 - [ ] **Browser Search Feature**:
   - [ ] Contextual search within webpage content
   - [ ] Semantic search over extracted content
@@ -388,15 +437,30 @@ Here’s what’s coming up next in **Ollama Client**—grouped by priority:
 
 ### File Upload & Processing
 
-- [ ] Plain text-based formats: `.txt`, `.md`, `.js`, `.ts` (direct read)
-- [ ] Support `.pdf` uploads via **pdf.js**
-- [ ] Image OCR via **Tesseract.js** (lazy-loaded)
-- [ ] Optional future support: `.docx` (via `mammoth.js`) etc.
+- [x] **Text File Support** (Beta v0.3.0 - Completed):
+  - [x] Plain text-based formats: `.txt`, `.md` and more.
+  - [x] Direct UTF-8 reading
+- [x] **PDF Support** (Beta v0.3.0 - Completed):
+  - [x] Full text extraction via **pdf.js**
+  - [x] Multi-page document support
+- [x] **DOCX Support** (Beta v0.3.0 - Completed):
+  - [x] Extract text from Word documents via **mammoth.js**
+  - [x] Handle formatting and structure
+- [x] **Auto-Embedding** (Beta v0.3.0 - Completed):
+  - [x] Automatic chunking with configurable strategies
+  - [x] Background embedding generation via port messaging
+  - [x] Progress tracking with real-time updates
+  - [x] Batch processing for performance
+- [x] **File Upload Settings** (Beta v0.3.0 - Completed):
+  - [x] Configurable max file size
+  - [x] Auto-embed toggle
+  - [x] Embedding batch size configuration
+- [ ] Image OCR via **Tesseract.js** (Deferred - CSP issues in Manifest V3)
 
 ### UX & Metrics Enhancements
 
 - [ ] Track **Per-Session Token Usage** and display in chat metadata (duration, token count)
-- [ ] Enable **Semantic Chat Search / Filter** once embeddings are in place
+- [x] Enable **Semantic Chat Search / Filter** once embeddings are in place
 - [x] Add **Export/Import UI Buttons** in chat selector ui
 
 ---

package.json

@@ -1,7 +1,7 @@
 {
   "name": "ollama-client",
   "displayName": "Ollama Client - Chat with Local LLM Models",
-  "version": "0.2.6",
+  "version": "0.3.1",
   "description": "Privacy-first Ollama Chrome extension to chat with local AI models like LLaMA, Mistral, Gemma — fully offline.",
   "author": "Shishir Chaurasiya",
   "keywords": [
@@ -96,6 +96,7 @@
     "highlight.js": "^11.11.1",
     "html2pdf.js": "^0.11.1",
     "lucide-react": "0.474.0",
+    "mammoth": "^1.6.0",
     "markdown-it": "^14.1.0",
     "markdown-it-container": "^4.0.0",
     "markdown-it-copy-code": "^0.1.3",
@@ -106,6 +107,7 @@
     "markdown-it-sub": "^2.0.0",
     "markdown-it-sup": "^2.0.0",
     "markdown-it-task-lists": "^2.1.1",
+    "pdfjs-dist": "^4.0.379",
     "plasmo": "0.90.5",
     "react": "18.2.0",
     "react-dom": "18.2.0",