campaign_connector/docs/enhanced-conversations-readme.md

# Enhanced Conversations - WhatsApp-Style Messaging Interface

## 🎯 Overview

The Enhanced Conversations feature transforms the SMS Campaign Manager into a WhatsApp-style messaging interface with real-time bidirectional sync, message status tracking, and advanced conversation management.

## ✨ Key Features

### 🔄 **Bidirectional SMS Sync**
- Automatically syncs SMS messages from your Android device
- Real-time message updates via WebSocket connection
- Support for both campaign and manual messages
- Smart conversation threading by phone number

### 📱 **WhatsApp-Style Interface**
- Clean, modern messaging UI with message bubbles
- Contact avatars with initials
- Message timestamps and status indicators
- Scrollable message history with pagination
- Real-time typing and connection status

### 📊 **Message Status Tracking**
- **Pending** ⏳ - Message queued for sending
- **Sent** ✓ - Successfully sent via Termux API
- **Delivered** ✓✓ - Confirmed delivery (when available)
- **Failed** ❌ - Send attempt failed

### 🌟 **Advanced Conversation Management**
- Star/unstar important conversations
- Mark conversations as read/unread
- Search conversations by name or phone number
- Filter by status (All, Unread, Starred)
- Contact name resolution from phone contacts

### ⚡ **Real-Time Updates**
- WebSocket-powered live messaging
- Instant message delivery notifications
- Automatic conversation sync
- Connection status indicators

## 🏗️ Architecture

### Backend Components

```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Main App      │    │  Sync Service    │    │ Android Device  │
│   (Flask)       │◄──►│  (Background)    │◄──►│ (Termux API)    │
│                 │    │                  │    │                 │
│ • API Routes    │    │ • SMS Pulling    │    │ • SMS History   │
│ • WebSocket     │    │ • Message Queue  │    │ • Contact Names │
│ • Database      │    │ • Status Updates │    │ • Send SMS      │
└─────────────────┘    └──────────────────┘    └─────────────────┘
```

### Database Schema

**Enhanced Conversations Table:**
- `is_starred` - Boolean flag for important conversations
- `contact_name` - Resolved contact name from phone
- `last_sync_timestamp` - Track sync progress
- `total_message_count` - Message count optimization

**Enhanced Messages Table:**
- `status` - Message delivery status
- `direction` - Inbound/outbound classification
- `timestamp` - Unix timestamp for sorting
- `external_message_id` - Phone's SMS ID for deduplication
- `sync_status` - Sync state tracking

## 🚀 Installation & Setup

### 1. Quick Setup

```bash
# Run the automated deployment
./scripts/deploy_enhanced_conversations.sh
```

### 2. Manual Setup

```bash
# 1. Install dependencies
cd src && pip install -r requirements.txt

# 2. Run database migration
python scripts/migrate_conversations_db.py

# 3. Integrate with main app
python scripts/integrate_enhanced_conversations.py

# 4. Update Android Termux API server
scp android/termux-sms-api-server.py android-dev@your-phone-ip:~/projects/sms-campaign-manager/

# 5. Restart services
docker-compose restart  # OR python src/app.py
```

### 3. Android Device Setup

The enhanced system requires updated Termux API endpoints:

```bash
# On your Android device (via SSH)
cd ~/projects/sms-campaign-manager
pkill -f termux-sms-api-server.py
python termux-sms-api-server.py &
```

## 🎮 Usage Guide

### Starting a Conversation

1. **From Campaign**: Conversations automatically created when sending campaign messages
2. **Manual Sync**: Click the sync button to pull message history from phone
3. **Direct Access**: Navigate to Conversations tab in dashboard

### Messaging Interface

```
┌─────────────────────────────────────────────────────────────────┐
│ [★] Contact Name              🔄 Sync    [Last seen: 2m ago]     │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Hello! How can I help?                               [9:15 AM] │
│      [You]: Thanks for reaching out ✓✓               [9:16 AM]  │
│                                                                 │
├─────────────────────────────────────────────────────────────────┤
│ Type a message...                                     [Send] 📤  │
└─────────────────────────────────────────────────────────────────┘
```

### Message Status Indicators

- ⏳ **Pending** - Message is being processed
- ✓ **Sent** - Successfully sent to phone
- ✓✓ **Delivered** - Confirmed delivery (when available)
- ❌ **Failed** - Send attempt failed

### Conversation Management

**Filtering:**
- **All** - Show all conversations
- **Unread** - Only conversations with unread messages
- **Starred** - Important/flagged conversations

**Actions:**
- **Star/Unstar** - Mark conversations as important
- **Mark as Read** - Clear unread indicators
- **Sync History** - Pull latest messages from phone
- **Search** - Find conversations by name or content

## 🔧 API Reference

### Enhanced Endpoints

```http
# Get conversations with enhanced data
GET /api/conversations/enhanced?search=john&starred=true

# Get paginated messages
GET /api/conversations/{id}/messages?page=1&per_page=50

# Send message
POST /api/conversations/{id}/send
Content-Type: application/json
{
  "message": "Hello there!"
}

# Toggle star status
PUT /api/conversations/{id}/star

# Mark as read
PUT /api/conversations/{id}/mark-read

# Manual sync
POST /api/conversations/{id}/sync
POST /api/conversations/sync-all

# Get statistics
GET /api/conversations/stats
```

### WebSocket Events

**Client → Server:**
```javascript
socket.emit('join_conversation', { conversation_id: 'conv_123' });
socket.emit('send_message', { 
  conversation_id: 'conv_123', 
  phone: '5551234567',
  message: 'Hello!' 
});
socket.emit('sync_conversation', { conversation_id: 'conv_123' });
```

**Server → Client:**
```javascript
socket.on('new_message', (data) => { /* Handle new message */ });
socket.on('message_status_update', (data) => { /* Update status */ });
socket.on('conversation_update', (data) => { /* Update conversation */ });
socket.on('sync_status', (data) => { /* Sync progress */ });
```

## 🧪 Testing

### Run Test Suite

```bash
# Full test suite
python scripts/test_enhanced_conversations.py

# Verbose output
python scripts/test_enhanced_conversations.py --verbose

# Custom endpoints
python scripts/test_enhanced_conversations.py --base-url http://localhost:5000 --phone-ip 192.168.1.100
```

### Manual Testing Checklist

- [ ] 🔌 WebSocket connection establishes
- [ ] 📱 Android device API responds
- [ ] 📤 Messages send successfully
- [ ] 📥 Incoming messages appear in real-time
- [ ] ⭐ Star/unstar functionality works
- [ ] 🔍 Search and filters work
- [ ] 📊 Status indicators update correctly
- [ ] 🔄 Manual sync pulls message history
- [ ] 📱 Contact names resolve from phone

## 🐛 Troubleshooting

### Common Issues

**WebSocket Connection Fails:**
```bash
# Check if Socket.IO is installed
pip show flask-socketio

# Test WebSocket endpoint
curl http://localhost:5000/socket.io/?EIO=4&transport=polling
```

**Android API Not Responding:**
```bash
# Test connectivity
ping your-phone-ip

# Check Termux API server
curl http://your-phone-ip:5001/health

# Restart server on Android
ssh android-dev "pkill -f termux-sms-api && python ~/projects/sms-campaign-manager/termux-sms-api-server.py &"
```

**Messages Not Syncing:**
```bash
# Check sync service logs
docker-compose logs -f sms-campaign

# Manual database inspection
sqlite3 data/campaign.db "SELECT * FROM messages ORDER BY timestamp DESC LIMIT 10;"

# Test SMS history endpoint
curl "http://your-phone-ip:5001/api/sms/history?limit=5"
```

**Database Errors:**
```bash
# Re-run migration
python scripts/migrate_conversations_db.py

# Check schema
sqlite3 data/campaign.db ".schema conversations"

# Verify permissions
ls -la data/campaign.db
```

### Debug Mode

Enable debug logging:

```python
# In app.py
logging.basicConfig(level=logging.DEBUG)

# In JavaScript console
localStorage.debug = 'socket.io-client:*';
```

## 📈 Performance Optimization

### Database Optimization

- **Indexes**: Automatic indexes on conversation_id, timestamp, status
- **Pagination**: Messages loaded in chunks of 50
- **Caching**: Conversation list cached in memory

### Network Optimization

- **WebSocket**: Persistent connection reduces HTTP overhead
- **Compression**: Message payloads automatically compressed
- **Rate Limiting**: 2-second delay between SMS sends

### Memory Management

- **Connection Pooling**: SQLite WAL mode for concurrent access
- **Message Cleanup**: Old messages archived after 30 days
- **Client-side**: Virtual scrolling for large message lists

## 🔒 Security Considerations

### Authentication

- **HMAC Signatures**: All Android API calls signed
- **Whitelisted Commands**: Only approved Termux commands allowed
- **Rate Limiting**: Prevents SMS spam and abuse

### Data Protection

- **Local Database**: Messages stored locally, not in cloud
- **Encrypted Transport**: HTTPS/WSS for all communications
- **Permission Model**: Respects Android SMS permissions

## 🤝 Contributing

### Development Setup

```bash
# Clone and setup
git clone <repository>
cd sms-campaign-manager

# Install development dependencies
pip install -r src/requirements.txt
pip install -r requirements-dev.txt

# Run in development mode
python src/app.py

# Run tests
python scripts/test_enhanced_conversations.py --verbose
```

### Architecture Guidelines

- **Backend**: Python with Flask, SQLite, asyncio
- **Frontend**: Vanilla JavaScript with Alpine.js
- **Real-time**: WebSocket via Socket.IO
- **Styling**: Tailwind CSS utility classes
- **Testing**: Python unittest + manual browser testing

## 📚 Additional Resources

- [Termux API Documentation](https://wiki.termux.com/wiki/Termux:API)
- [Flask-SocketIO Guide](https://flask-socketio.readthedocs.io/)
- [Alpine.js Documentation](https://alpinejs.dev/)
- [SMS Campaign Manager Wiki](../docs/)

---

**Built with ❤️ for efficient SMS campaign management and real-time messaging.**