freealberta/map/build-nocodb.md

# NocoDB Automation Script Development Summary

## Overview
This document summarizes the development of an automated NocoDB table creation script (`build-nocodb.sh`) for the Map Viewer project. The script automates the creation of three required tables: `locations`, `login`, and `settings` with proper schemas and default data.

## Project Requirements
Based on the README.md analysis, the project needed:
- **locations** table: Main map data storage
- **login** table: User authentication 
- **settings** table: System configuration and QR codes
- Default admin user and start location records
- Idempotent script (safe to re-run)

## NocoDB API Research

### API Versions
- **v1 API**: `/api/v1/` - Legacy, limited functionality
- **v2 API**: `/api/v2/` - Modern, full-featured (recommended)

### Key API Endpoints Discovered

#### Base/Project Management
```
GET /api/v2/meta/bases          # List all bases
POST /api/v2/meta/bases         # Create new base
GET /api/v2/meta/bases/{id}     # Get base details
```

#### Table Management
```
GET /api/v2/meta/bases/{base_id}/tables     # List tables in base
POST /api/v2/meta/bases/{base_id}/tables    # Create table
GET /api/v2/meta/bases/{base_id}/tables/{table_id}  # Get table details
```

#### Record Management
```
GET /api/v2/tables/{table_id}/records       # List records
POST /api/v2/tables/{table_id}/records      # Create record
PUT /api/v2/tables/{table_id}/records/{record_id}  # Update record
```

### Authentication
All API calls require the `xc-token` header:
```bash
curl -H "xc-token: YOUR_TOKEN" -H "Content-Type: application/json"
```

## Table Schemas Implemented

### 1. Locations Table
Primary table for map data storage:
```json
{
  "table_name": "locations",
  "columns": [
    {"column_name": "id", "uidt": "ID", "pk": true, "ai": true},
    {"column_name": "title", "uidt": "SingleLineText"},
    {"column_name": "description", "uidt": "LongText"},
    {"column_name": "category", "uidt": "SingleSelect", "colOptions": {
      "options": [
        {"title": "Important", "color": "#ff0000"},
        {"title": "Event", "color": "#00ff00"},
        {"title": "Business", "color": "#0000ff"},
        {"title": "Other", "color": "#ffff00"}
      ]
    }},
    {"column_name": "geo_location", "uidt": "LongText"},
    {"column_name": "latitude", "uidt": "Decimal"},
    {"column_name": "longitude", "uidt": "Decimal"},
    {"column_name": "address", "uidt": "LongText"},
    {"column_name": "contact_info", "uidt": "LongText"},
    {"column_name": "created_at", "uidt": "DateTime"},
    {"column_name": "updated_at", "uidt": "DateTime"}
  ]
}
```

### 2. Login Table
User authentication table:
```json
{
  "table_name": "login",
  "columns": [
    {"column_name": "id", "uidt": "ID", "pk": true, "ai": true},
    {"column_name": "username", "uidt": "SingleLineText", "rqd": true},
    {"column_name": "email", "uidt": "Email", "rqd": true},
    {"column_name": "password", "uidt": "SingleLineText", "rqd": true},
    {"column_name": "admin", "uidt": "Checkbox"},
    {"column_name": "active", "uidt": "Checkbox"},
    {"column_name": "created_at", "uidt": "DateTime"},
    {"column_name": "last_login", "uidt": "DateTime"}
  ]
}
```

### 3. Settings Table
System configuration with QR code support:
```json
{
  "table_name": "settings",
  "columns": [
    {"column_name": "id", "uidt": "ID", "pk": true, "ai": true},
    {"column_name": "key", "uidt": "SingleLineText", "rqd": true},
    {"column_name": "title", "uidt": "SingleLineText"},
    {"column_name": "geo_location", "uidt": "LongText"},
    {"column_name": "latitude", "uidt": "Decimal"},
    {"column_name": "longitude", "uidt": "Decimal"},
    {"column_name": "zoom", "uidt": "Number"},
    {"column_name": "category", "uidt": "SingleSelect", "colOptions": {
      "options": [
        {"title": "system_setting", "color": "#4CAF50"},
        {"title": "user_setting", "color": "#2196F3"},
        {"title": "app_config", "color": "#FF9800"}
      ]
    }},
    {"column_name": "updated_by", "uidt": "SingleLineText"},
    {"column_name": "updated_at", "uidt": "DateTime"},
    {"column_name": "qr_code_1_url", "uidt": "URL"},
    {"column_name": "qr_code_1_label", "uidt": "SingleLineText"},
    {"column_name": "qr_code_1_image", "uidt": "Attachment"},
    {"column_name": "qr_code_2_url", "uidt": "URL"},
    {"column_name": "qr_code_2_label", "uidt": "SingleLineText"},
    {"column_name": "qr_code_2_image", "uidt": "Attachment"},
    {"column_name": "qr_code_3_url", "uidt": "URL"},
    {"column_name": "qr_code_3_label", "uidt": "SingleLineText"},
    {"column_name": "qr_code_3_image", "uidt": "Attachment"}
  ]
}
```

## NocoDB Column Types (UIdt)
Discovered column types and their usage:
- `ID` - Auto-incrementing primary key
- `SingleLineText` - Short text field
- `LongText` - Multi-line text area
- `Email` - Email validation
- `URL` - URL validation
- `Decimal` - Decimal numbers
- `Number` - Integer numbers
- `DateTime` - Date and time
- `Checkbox` - Boolean true/false
- `SingleSelect` - Dropdown with predefined options
- `Attachment` - File upload field

## Script Development Process

### Initial Implementation
1. Created basic structure with environment variable loading
2. Implemented API connectivity testing
3. Added base/project creation functionality
4. Created table creation functions

### Key Challenges Solved

#### 1. Environment Variable Loading
**Issue**: Standard `source .env` wasn't exporting variables
**Solution**: Use `set -a; source .env; set +a` pattern
```bash
set -a        # Auto-export all variables
source .env   # Load environment file
set +a        # Disable auto-export
```

#### 2. API Version Compatibility
**Issue**: Mixed v1/v2 endpoint usage causing errors
**Solution**: Standardized on v2 API with proper URL construction
```bash
BASE_URL=$(echo "$NOCODB_API_URL" | sed 's|/api/v1||')
API_BASE_V2="${BASE_URL}/api/v2"
```

#### 3. Duplicate Table Error
**Issue**: Script failed when tables already existed
**Solution**: Added idempotent table checking
```bash
get_table_id_by_name() {
    local base_id=$1
    local table_name=$2
    
    # Check if table exists by name
    local tables_response
    tables_response=$(make_api_call "GET" "/meta/bases/$base_id/tables" "" "Fetching tables")
    
    # Parse JSON to find table ID
    local table_id
    table_id=$(echo "$tables_response" | grep -o '"id":"[^"]*","table_name":"'"$table_name"'"' | grep -o '"id":"[^"]*"' | head -1 | sed 's/"id":"//;s/"//')
    
    if [ -n "$table_id" ]; then
        echo "$table_id"
        return 0
    else
        return 1
    fi
}
```

#### 4. JSON Response Parsing
**Issue**: Complex JSON parsing for table IDs
**Solution**: Used grep with regex patterns
```bash
# Extract table ID from JSON response
table_id=$(echo "$response" | grep -o '"id":"[^"]*"' | head -1 | cut -d'"' -f4)
```

## Default Data Records

### Admin User
```json
{
  "username": "admin",
  "email": "admin@example.com", 
  "password": "changeme123",
  "admin": true,
  "active": true,
  "created_at": "2025-07-05 12:00:00"
}
```

### Start Location Setting
```json
{
  "key": "start_location",
  "title": "Map Start Location",
  "geo_location": "53.5461;-113.4938",
  "latitude": 53.5461,
  "longitude": -113.4938,
  "zoom": 11,
  "category": "system_setting",
  "updated_by": "system",
  "updated_at": "2025-07-05 12:00:00"
}
```

## Error Handling Patterns

### API Call Wrapper
```bash
make_api_call() {
    local method=$1
    local endpoint=$2
    local data=$3
    local description=$4
    local api_version=${5:-"v2"}
    
    # Construct full URL
    if [[ "$api_version" == "v1" ]]; then
        full_url="$API_BASE_V1$endpoint"
    else
        full_url="$API_BASE_V2$endpoint"
    fi
    
    # Make request with timeout
    response=$(curl -s -w "%{http_code}" -X "$method" \
               -H "xc-token: $NOCODB_API_TOKEN" \
               -H "Content-Type: application/json" \
               --max-time 30 \
               -d "$data" \
               "$full_url" 2>/dev/null)
    
    # Parse HTTP code and response
    http_code="${response: -3}"
    response_body="${response%???}"
    
    # Check for success
    if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then
        echo "$response_body"
        return 0
    else
        print_error "API call failed: $http_code - $response_body"
        return 1
    fi
}
```

## Final Script Features

### Idempotent Operation
- Checks for existing base/project
- Validates table existence before creation
- Uses existing table IDs when found
- Safe to run multiple times

### Robust Error Handling
- Network timeout protection
- HTTP status code validation
- JSON parsing error handling
- Colored output for status messages

### Environment Integration
- Loads configuration from `.env` file
- Supports custom default coordinates
- Validates required variables

## Usage Instructions

1. **Setup Environment**:
   ```bash
   # Update .env with your NocoDB details
   NOCODB_API_URL=https://your-nocodb.com/api/v1
   NOCODB_API_TOKEN=your_token_here
   ```

2. **Run Script**:
   ```bash
   chmod +x build-nocodb.sh
   ./build-nocodb.sh
   ```

3. **Post-Setup**:
   - Update `.env` with generated table URLs
   - Change default admin password
   - Verify tables in NocoDB interface

## Lessons Learned

1. **API Documentation**: Always verify API endpoints with actual testing
2. **JSON Parsing**: Shell-based JSON parsing requires careful regex patterns
3. **Idempotency**: Essential for automation scripts in production
4. **Error Handling**: Comprehensive error handling prevents silent failures
5. **Environment Variables**: Proper loading patterns are crucial for script reliability

## Future Enhancements

- Add support for custom table schemas via configuration
- Implement data migration features
- Add backup/restore functionality
- Support for multiple environment configurations
- Integration with CI/CD pipelines

## Script Updates - July 2025

### Column Type Improvements
Updated the build-nocodb.sh script to use proper NocoDB column types based on the official documentation:

#### Locations Table Updates
- **`geo_location`**: Changed from `LongText` to `GeoData` (proper geographic data type)
- **`latitude`**: Added precision (10) and scale (8) for proper decimal handling
- **`longitude`**: Added precision (11) and scale (8) for proper decimal handling
- **`phone`**: Changed from `SingleLineText` to `PhoneNumber` (proper phone validation)
- **`email`**: Using `Email` type for proper email validation
- **Updated field names**: Added proper fields from README.md:
  - `first_name`, `last_name` (SingleLineText)
  - `unit_number` (SingleLineText)  
  - `support_level` (SingleSelect with colors: 1=Green, 2=Yellow, 3=Orange, 4=Red)
  - `sign` (Checkbox)
  - `sign_size` (SingleSelect: Regular, Large, Unsure)
  - `notes` (LongText)
  - `address` (SingleLineText instead of LongText)

#### Login Table Updates
- **Simplified structure**: Removed username/password fields per README.md specification
- **Core fields**: `email` (Email), `name` (SingleLineText), `admin` (Checkbox)
- **Authentication note**: This is a simplified table - proper authentication should be implemented separately

#### Settings Table Updates
- **`geo_location`**: Changed from `LongText` to `GeoData` for proper geographic data handling
- **`latitude`/`longitude`**: Added precision and scale parameters
- **`value`**: Added missing `value` field from README.md specification
- **QR Code fields**: Simplified to just attachment fields (removed URL/label fields not in README.md)

### Benefits of Proper Column Types

1. **GeoData Type**: 
   - Proper latitude;longitude format validation
   - Better integration with mapping libraries
   - Consistent data storage format

2. **PhoneNumber Type**:
   - Built-in phone number validation
   - Proper formatting and display
   - International number support

3. **Email Type**:
   - Email format validation
   - Prevents invalid email addresses
   - Better UI experience

4. **Decimal Precision**:
   - Latitude: 10 digits, 8 decimal places (±90.12345678)
   - Longitude: 11 digits, 8 decimal places (±180.12345678)
   - Provides GPS-level precision for mapping

5. **SingleSelect with Colors**:
   - Support Level: Color-coded options for visual feedback
   - Sign Size: Consistent option selection
   - Category: Organized classification system

### Backward Compatibility
The script maintains backward compatibility while using proper column types. Existing data migration may be needed if upgrading from the old schema.

## Walk Sheet Implementation Overhaul - July 2025

### Overview
The walk sheet system has been completely overhauled to simplify QR code handling and improve mobile usability. The new approach stores only text configuration and generates QR codes on-demand.

### Key Changes Made

#### 1. Database Schema Simplification
- **Removed**: `qr_code_1_image`, `qr_code_2_image`, `qr_code_3_image` attachment fields
- **Kept**: Only text fields for URLs and labels:
  - `walk_sheet_title`, `walk_sheet_subtitle`, `walk_sheet_footer`
  - `qr_code_1_url`, `qr_code_1_label`
  - `qr_code_2_url`, `qr_code_2_label`
  - `qr_code_3_url`, `qr_code_3_label`

#### 2. Backend API Updates
- **GET `/api/admin/walk-sheet-config`**: Returns only text configuration
- **POST `/api/admin/walk-sheet-config`**: Saves only text fields
- **Removed**: All QR code upload/storage logic
- **Kept**: Local QR generation via `/api/qr` endpoint for preview/print

#### 3. Frontend Improvements
- **Simplified JavaScript**: Removed `storedQRCodes` logic and image upload handling
- **Better Mobile Support**: Responsive layout with stacked preview on mobile
- **Larger Preview**: Increased from 50% to 75% scale on desktop
- **Real-time Preview**: QR codes generated on-the-fly using canvas

#### 4. CSS Redesign
- **Desktop**: 40/60 split (config/preview) for better preview visibility
- **Mobile**: Stacked layout with horizontal scroll for preview
- **Improved Scaling**: Better touch targets and spacing
- **Professional Styling**: Enhanced typography and visual hierarchy

### Benefits of New Approach
1. **Simpler**: No file storage complexity
2. **Faster**: No upload/download of images
3. **Flexible**: QR codes always reflect current URLs
4. **Cleaner**: Database only stores configuration text
5. **Scalable**: No storage concerns for QR images
6. **Mobile-Friendly**: Better responsive design

### Migration Notes
- Existing QR image data can be ignored (will be regenerated)
- Text configuration will be preserved
- No data loss as QR codes are generated from URLs
- Safe to run build script multiple times

---
*Generated: July 5, 2025*
*Script Version: Column Type Optimized*