freealberta/map/instruct/LISTMONK_INTEGRATION_GUIDE.md

# Listmonk Integration Guide

This guide covers the complete setup and usage of the Listmonk integration for the Map application.

## Overview

The Map application integrates seamlessly with [Listmonk](https://listmonk.app/), a self-hosted newsletter and mailing list manager, to provide advanced email marketing capabilities. This integration enables:

- **Real-time subscriber synchronization**
- **Automated list segmentation** 
- **Professional email campaigns**
- **Advanced analytics and reporting**
- **Bi-directional data sync**

## Prerequisites

1. **Listmonk Instance**: You need a running Listmonk instance (typically deployed via Docker Compose)
2. **Admin Access**: Listmonk admin credentials for API access
3. **Network Connectivity**: Map application must be able to reach Listmonk instance
4. **Environment Configuration**: Proper `.env` setup

## Quick Setup

### 1. Configure Environment Variables

Add these variables to your `.env` file:

```env
# Listmonk Integration
LISTMONK_URL=http://listmonk:9000
LISTMONK_USERNAME=admin
LISTMONK_PASSWORD=your-secure-password
LISTMONK_ENABLED=true
LISTMONK_SYNC_ON_STARTUP=true
LISTMONK_AUTO_CREATE_LISTS=true
```

**Important Notes:**
- Use `http://listmonk:9000` for Docker Compose deployments (service name)
- Use `http://localhost:9000` for local development
- Ensure the password matches your Listmonk admin password

### 2. Restart the Application

```bash
docker-compose restart map-viewer
```

### 3. Verify Integration

1. **Check Admin Panel**: Go to `/admin.html` and look for the "Listmonk Integration" section
2. **Status Indicator**: Green status means successful connection
3. **View Logs**: Check terminal for sync messages
4. **Listmonk Dashboard**: Verify lists were created in Listmonk

## Automated List Management

The integration automatically creates and maintains these subscriber lists:

### Core Lists
- **All Locations** - Every location with contact information
- **All Users** - Every registered Map application user

### Support Level Lists
- **Support Level 1** - Strongest supporters (Green markers)
- **Support Level 2** - Moderate supporters (Yellow markers)  
- **Support Level 3** - Weak supporters (Orange markers)
- **Support Level 4** - Opponents (Red markers)

### Sign Status Lists
- **Has Sign** - Locations with campaign signs
- **No Sign** - Locations without signs

### Combined Segmentation Lists
- **Support 1 with Signs** - Strong supporters who have signs
- **Support 2 with Signs** - Moderate supporters who have signs
- **Support 3 with Signs** - Weak supporters who have signs
- **Support 4 with Signs** - Opponents who have signs (for tracking)
- **Support 1-3** - All positive supporters (Levels 1, 2, and 3)
- **Sign Requests** - Supporters without signs (potential sign placements)

## Real-Time Sync Features

### Automatic Synchronization
- **New Location Added**: Instantly added to appropriate lists
- **Location Updated**: Support level or sign status changes trigger list updates
- **Location Deleted**: Removed from all lists
- **User Created**: Added to "All Users" list
- **User Deleted**: Removed from all lists

### Bi-Directional Sync
- **Unsubscribes**: When users unsubscribe in Listmonk, their records are updated in the Map database
- **Email Updates**: Email address changes sync both ways
- **List Management**: Changes in Listmonk are reflected in Map data

## Admin Management Features

### Status Dashboard
- **Connection Status**: Live indicator (🟢 Connected, 🔴 Error, ⚪ Disabled)
- **Sync Statistics**: Total subscribers, lists, and sync operations
- **Last Sync Times**: When each data type was last synchronized
- **Error Notifications**: Pop-up alerts for sync failures

### Manual Operations
- **Full Resync**: Completely rebuild all lists from current data
- **Selective Sync**: Sync only locations or users
- **Test Connection**: Verify Listmonk connectivity
- **Reinitialize**: Delete and recreate all lists
- **Progress Tracking**: Visual progress bars for bulk operations

### Bulk Sync Controls
```javascript
// Available admin actions
POST /api/listmonk/sync/full        // Complete resync
POST /api/listmonk/sync/locations   // Sync locations only  
POST /api/listmonk/sync/users       // Sync users only
POST /api/listmonk/test-connection  // Test connectivity
POST /api/listmonk/reinitialize     // Recreate lists
```

## Error Handling & Troubleshooting

### Common Issues

#### Connection Failed
**Symptoms**: Red status indicator, "Connection failed" in logs
**Solutions**:
1. Verify `LISTMONK_URL` is correct
2. Check Listmonk is running: `docker-compose ps`
3. Test network connectivity: `curl http://listmonk:9000`
4. Verify credentials in Listmonk admin panel

#### Sync Errors  
**Symptoms**: Yellow status indicator, specific error messages in logs
**Solutions**:
1. Check Listmonk API limits and performance
2. Verify subscriber data format (valid emails, etc.)
3. Review Listmonk logs: `docker-compose logs listmonk`
4. Try manual resync from admin panel

#### Missing Lists
**Symptoms**: Lists not appearing in Listmonk
**Solutions**:
1. Ensure `LISTMONK_AUTO_CREATE_LISTS=true`
2. Restart application to trigger initialization
3. Use "Reinitialize" button in admin panel
4. Check Listmonk permissions and API access

#### Slow Sync Performance
**Symptoms**: Long delays during sync operations
**Solutions**:
1. Monitor Listmonk server resources
2. Check network latency between services
3. Consider reducing sync frequency for large datasets
4. Review database performance (both NocoDB and Listmonk)

### Debug Logging

Enable detailed logging by setting:
```env
NODE_ENV=development
```

This provides verbose sync operation details in the terminal.

## Email Campaign Workflow

### 1. Segment Selection
Use the automatically created lists in Listmonk to target specific audience segments:
- **Fundraising**: Target "Support Level 1" for donation requests
- **Volunteer Recruitment**: Target "Support 1-3" for volunteer opportunities  
- **Sign Placement**: Target "Sign Requests" for new sign installations
- **Opposition Research**: Monitor "Support Level 4" engagement

### 2. Campaign Creation
Create campaigns in Listmonk using the segmented lists:
1. Go to Listmonk admin panel
2. Create new campaign
3. Select appropriate subscriber lists
4. Design email content
5. Schedule or send immediately

### 3. Analytics & Follow-up
- **Track Opens**: Monitor engagement by support level
- **Click Tracking**: Measure action rates across segments
- **Unsubscribe Monitoring**: Identify potential support level changes
- **A/B Testing**: Test messaging effectiveness across segments

## Advanced Configuration

### Custom List Creation
To create additional custom lists, modify the Listmonk service configuration:

```javascript
// app/services/listmonk.js
const customLists = [
    {
        name: 'High Value Voters',
        description: 'Support Level 1 & 2 with voting history',
        // Add custom segmentation logic
    }
];
```

### Sync Frequency Tuning
Adjust sync timing based on your needs:

```env
# Sync immediately on data changes (default: true)
LISTMONK_REALTIME_SYNC=true

# Startup sync (default: true) 
LISTMONK_SYNC_ON_STARTUP=true

# Batch size for large syncs (default: 100)
LISTMONK_BATCH_SIZE=50
```

### Webhook Integration
For advanced use cases, consider setting up Listmonk webhooks to trigger Map application updates:

```env
# Enable webhook endpoint
LISTMONK_WEBHOOK_ENABLED=true
LISTMONK_WEBHOOK_SECRET=your-webhook-secret
```

## Security Considerations

### Credentials Management
- **Store securely**: Use environment variables, never commit passwords
- **Rotate regularly**: Change Listmonk passwords periodically
- **Limit access**: Create dedicated API user with minimal required permissions
- **Network security**: Use HTTPS in production, restrict network access

### Data Privacy
- **GDPR Compliance**: Ensure proper consent and unsubscribe handling
- **Data Minimization**: Only sync necessary contact information
- **Retention Policies**: Configure appropriate data retention in Listmonk
- **Audit Logging**: Monitor access and changes to subscriber data

## Integration Monitoring

### Health Checks
The system provides several monitoring endpoints:

```javascript
GET /api/listmonk/status        // Connection and sync status
GET /api/listmonk/lists         // List information and counts
GET /health                     // Overall application health
```

### Performance Metrics
Monitor these key metrics:
- **Sync Duration**: Time to complete full synchronization
- **Error Rate**: Percentage of failed sync operations  
- **List Growth**: Subscriber count changes over time
- **API Response Time**: Listmonk API performance

### Alerting
Set up monitoring for:
- **Connection failures** (implement external monitoring)
- **Sync errors** (check application logs)
- **List count discrepancies** (compare Map vs Listmonk counts)
- **Performance degradation** (sync time increases)

## Backup & Recovery

### Data Backup
- **Listmonk Data**: Regular database backups of Listmonk
- **Map Data**: NocoDB backup includes all source data
- **Configuration**: Version control `.env` files (without secrets)

### Disaster Recovery
In case of data loss:
1. **Restore Listmonk**: From database backup
2. **Reinitialize**: Use admin panel "Reinitialize" function  
3. **Full Resync**: Rebuild all lists from Map data
4. **Verify**: Check list counts and subscriber data

## Best Practices

### Campaign Management
- **Consistent Messaging**: Maintain brand voice across all campaigns
- **Segmentation Strategy**: Use meaningful, actionable segments
- **Testing**: A/B test subject lines and content
- **Compliance**: Follow CAN-SPAM and GDPR requirements
- **Analytics**: Track and analyze campaign performance

### Data Quality
- **Email Validation**: Verify email addresses before adding to lists
- **Duplicate Prevention**: System handles duplicates automatically
- **Regular Cleanup**: Remove bounced and inactive subscribers
- **Contact Updates**: Keep contact information current

### Performance Optimization
- **Batch Operations**: Use bulk sync for large data changes
- **Off-Peak Sync**: Schedule heavy operations during low-usage periods
- **Resource Monitoring**: Watch server resources during sync operations
- **Network Optimization**: Ensure reliable connectivity between services

## Support & Documentation

### Additional Resources
- **Listmonk Documentation**: https://listmonk.app/docs/
- **Map Application Docs**: See main README.md
- **API Reference**: Available at `/api` endpoints
- **Community Support**: GitHub issues and discussions

### Getting Help
If you encounter issues:
1. **Check Status**: Admin panel shows current integration status
2. **Review Logs**: Terminal output provides detailed error information
3. **Test Connection**: Use admin panel connection test
4. **Documentation**: Refer to this guide and Listmonk docs
5. **Community**: Ask questions in project GitHub issues

This integration significantly enhances the Map application's email marketing capabilities while maintaining data consistency and providing powerful segmentation options for targeted campaigns.