admin 776420b6b8 build shell script for NocoDB map data

2025-07-05 19:52:17 -06:00

12 KiB

Raw Blame History

NocoDB Automation Script Development Summary

Overview

This document summarizes the development of an automated NocoDB table creation script (build-nocodb.sh) for the Map Viewer project. The script automates the creation of three required tables: locations, login, and settings with proper schemas and default data.

Project Requirements

Based on the README.md analysis, the project needed:

locations table: Main map data storage
login table: User authentication
settings table: System configuration and QR codes
Default admin user and start location records
Idempotent script (safe to re-run)

NocoDB API Research

API Versions

v1 API: /api/v1/ - Legacy, limited functionality
v2 API: /api/v2/ - Modern, full-featured (recommended)

Key API Endpoints Discovered

Base/Project Management

GET /api/v2/meta/bases          # List all bases
POST /api/v2/meta/bases         # Create new base
GET /api/v2/meta/bases/{id}     # Get base details

Table Management

GET /api/v2/meta/bases/{base_id}/tables     # List tables in base
POST /api/v2/meta/bases/{base_id}/tables    # Create table
GET /api/v2/meta/bases/{base_id}/tables/{table_id}  # Get table details

Record Management

GET /api/v2/tables/{table_id}/records       # List records
POST /api/v2/tables/{table_id}/records      # Create record
PUT /api/v2/tables/{table_id}/records/{record_id}  # Update record

Authentication

All API calls require the xc-token header:

curl -H "xc-token: YOUR_TOKEN" -H "Content-Type: application/json"

Table Schemas Implemented

1. Locations Table

Primary table for map data storage:

{
  "table_name": "locations",
  "columns": [
    {"column_name": "id", "uidt": "ID", "pk": true, "ai": true},
    {"column_name": "title", "uidt": "SingleLineText"},
    {"column_name": "description", "uidt": "LongText"},
    {"column_name": "category", "uidt": "SingleSelect", "colOptions": {
      "options": [
        {"title": "Important", "color": "#ff0000"},
        {"title": "Event", "color": "#00ff00"},
        {"title": "Business", "color": "#0000ff"},
        {"title": "Other", "color": "#ffff00"}
      ]
    }},
    {"column_name": "geo_location", "uidt": "LongText"},
    {"column_name": "latitude", "uidt": "Decimal"},
    {"column_name": "longitude", "uidt": "Decimal"},
    {"column_name": "address", "uidt": "LongText"},
    {"column_name": "contact_info", "uidt": "LongText"},
    {"column_name": "created_at", "uidt": "DateTime"},
    {"column_name": "updated_at", "uidt": "DateTime"}
  ]
}

User authentication table:

{
  "table_name": "login",
  "columns": [
    {"column_name": "id", "uidt": "ID", "pk": true, "ai": true},
    {"column_name": "username", "uidt": "SingleLineText", "rqd": true},
    {"column_name": "email", "uidt": "Email", "rqd": true},
    {"column_name": "password", "uidt": "SingleLineText", "rqd": true},
    {"column_name": "admin", "uidt": "Checkbox"},
    {"column_name": "active", "uidt": "Checkbox"},
    {"column_name": "created_at", "uidt": "DateTime"},
    {"column_name": "last_login", "uidt": "DateTime"}
  ]
}

3. Settings Table

System configuration with QR code support:

{
  "table_name": "settings",
  "columns": [
    {"column_name": "id", "uidt": "ID", "pk": true, "ai": true},
    {"column_name": "key", "uidt": "SingleLineText", "rqd": true},
    {"column_name": "title", "uidt": "SingleLineText"},
    {"column_name": "geo_location", "uidt": "LongText"},
    {"column_name": "latitude", "uidt": "Decimal"},
    {"column_name": "longitude", "uidt": "Decimal"},
    {"column_name": "zoom", "uidt": "Number"},
    {"column_name": "category", "uidt": "SingleSelect", "colOptions": {
      "options": [
        {"title": "system_setting", "color": "#4CAF50"},
        {"title": "user_setting", "color": "#2196F3"},
        {"title": "app_config", "color": "#FF9800"}
      ]
    }},
    {"column_name": "updated_by", "uidt": "SingleLineText"},
    {"column_name": "updated_at", "uidt": "DateTime"},
    {"column_name": "qr_code_1_url", "uidt": "URL"},
    {"column_name": "qr_code_1_label", "uidt": "SingleLineText"},
    {"column_name": "qr_code_1_image", "uidt": "Attachment"},
    {"column_name": "qr_code_2_url", "uidt": "URL"},
    {"column_name": "qr_code_2_label", "uidt": "SingleLineText"},
    {"column_name": "qr_code_2_image", "uidt": "Attachment"},
    {"column_name": "qr_code_3_url", "uidt": "URL"},
    {"column_name": "qr_code_3_label", "uidt": "SingleLineText"},
    {"column_name": "qr_code_3_image", "uidt": "Attachment"}
  ]
}

NocoDB Column Types (UIdt)

Discovered column types and their usage:

ID - Auto-incrementing primary key
SingleLineText - Short text field
LongText - Multi-line text area
Email - Email validation
URL - URL validation
Decimal - Decimal numbers
Number - Integer numbers
DateTime - Date and time
Checkbox - Boolean true/false
SingleSelect - Dropdown with predefined options
Attachment - File upload field

Script Development Process

Initial Implementation

Created basic structure with environment variable loading
Implemented API connectivity testing
Added base/project creation functionality
Created table creation functions

Key Challenges Solved

1. Environment Variable Loading

Issue: Standard source .env wasn't exporting variables Solution: Use set -a; source .env; set +a pattern

set -a        # Auto-export all variables
source .env   # Load environment file
set +a        # Disable auto-export

2. API Version Compatibility

Issue: Mixed v1/v2 endpoint usage causing errors Solution: Standardized on v2 API with proper URL construction

BASE_URL=$(echo "$NOCODB_API_URL" | sed 's|/api/v1||')
API_BASE_V2="${BASE_URL}/api/v2"

3. Duplicate Table Error

Issue: Script failed when tables already existed Solution: Added idempotent table checking

get_table_id_by_name() {
    local base_id=$1
    local table_name=$2
    
    # Check if table exists by name
    local tables_response
    tables_response=$(make_api_call "GET" "/meta/bases/$base_id/tables" "" "Fetching tables")
    
    # Parse JSON to find table ID
    local table_id
    table_id=$(echo "$tables_response" | grep -o '"id":"[^"]*","table_name":"'"$table_name"'"' | grep -o '"id":"[^"]*"' | head -1 | sed 's/"id":"//;s/"//')
    
    if [ -n "$table_id" ]; then
        echo "$table_id"
        return 0
    else
        return 1
    fi
}

4. JSON Response Parsing

Issue: Complex JSON parsing for table IDs Solution: Used grep with regex patterns

# Extract table ID from JSON response
table_id=$(echo "$response" | grep -o '"id":"[^"]*"' | head -1 | cut -d'"' -f4)

Default Data Records

Admin User

{
  "username": "admin",
  "email": "admin@example.com", 
  "password": "changeme123",
  "admin": true,
  "active": true,
  "created_at": "2025-07-05 12:00:00"
}

Start Location Setting

{
  "key": "start_location",
  "title": "Map Start Location",
  "geo_location": "53.5461;-113.4938",
  "latitude": 53.5461,
  "longitude": -113.4938,
  "zoom": 11,
  "category": "system_setting",
  "updated_by": "system",
  "updated_at": "2025-07-05 12:00:00"
}

Error Handling Patterns

API Call Wrapper

make_api_call() {
    local method=$1
    local endpoint=$2
    local data=$3
    local description=$4
    local api_version=${5:-"v2"}
    
    # Construct full URL
    if [[ "$api_version" == "v1" ]]; then
        full_url="$API_BASE_V1$endpoint"
    else
        full_url="$API_BASE_V2$endpoint"
    fi
    
    # Make request with timeout
    response=$(curl -s -w "%{http_code}" -X "$method" \
               -H "xc-token: $NOCODB_API_TOKEN" \
               -H "Content-Type: application/json" \
               --max-time 30 \
               -d "$data" \
               "$full_url" 2>/dev/null)
    
    # Parse HTTP code and response
    http_code="${response: -3}"
    response_body="${response%???}"
    
    # Check for success
    if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then
        echo "$response_body"
        return 0
    else
        print_error "API call failed: $http_code - $response_body"
        return 1
    fi
}

Final Script Features

Idempotent Operation

Checks for existing base/project
Validates table existence before creation
Uses existing table IDs when found
Safe to run multiple times

Robust Error Handling

Network timeout protection
HTTP status code validation
JSON parsing error handling
Colored output for status messages

Environment Integration

Loads configuration from .env file
Supports custom default coordinates
Validates required variables

Usage Instructions

Setup Environment:

# Update .env with your NocoDB details
NOCODB_API_URL=https://your-nocodb.com/api/v1
NOCODB_API_TOKEN=your_token_here

Run Script:

chmod +x build-nocodb.sh
./build-nocodb.sh

Post-Setup:
- Update .env with generated table URLs
- Change default admin password
- Verify tables in NocoDB interface

Lessons Learned

API Documentation: Always verify API endpoints with actual testing
JSON Parsing: Shell-based JSON parsing requires careful regex patterns
Idempotency: Essential for automation scripts in production
Error Handling: Comprehensive error handling prevents silent failures
Environment Variables: Proper loading patterns are crucial for script reliability

Future Enhancements

Add support for custom table schemas via configuration
Implement data migration features
Add backup/restore functionality
Support for multiple environment configurations
Integration with CI/CD pipelines

Script Updates - July 2025

Column Type Improvements

Updated the build-nocodb.sh script to use proper NocoDB column types based on the official documentation:

Locations Table Updates

geo_location: Changed from LongText to GeoData (proper geographic data type)
latitude: Added precision (10) and scale (8) for proper decimal handling
longitude: Added precision (11) and scale (8) for proper decimal handling
phone: Changed from SingleLineText to PhoneNumber (proper phone validation)
email: Using Email type for proper email validation
Updated field names: Added proper fields from README.md:
- first_name, last_name (SingleLineText)
- unit_number (SingleLineText)
- support_level (SingleSelect with colors: 1=Green, 2=Yellow, 3=Orange, 4=Red)
- sign (Checkbox)
- sign_size (SingleSelect: Small, Medium, Large)
- notes (LongText)
- address (SingleLineText instead of LongText)

Simplified structure: Removed username/password fields per README.md specification
Core fields: email (Email), name (SingleLineText), admin (Checkbox)
Authentication note: This is a simplified table - proper authentication should be implemented separately

Settings Table Updates

geo_location: Changed from LongText to GeoData for proper geographic data handling
latitude/longitude: Added precision and scale parameters
value: Added missing value field from README.md specification
QR Code fields: Simplified to just attachment fields (removed URL/label fields not in README.md)

Benefits of Proper Column Types

GeoData Type:
- Proper latitude;longitude format validation
- Better integration with mapping libraries
- Consistent data storage format
PhoneNumber Type:
- Built-in phone number validation
- Proper formatting and display
- International number support
Email Type:
- Email format validation
- Prevents invalid email addresses
- Better UI experience
Decimal Precision:
- Latitude: 10 digits, 8 decimal places (±90.12345678)
- Longitude: 11 digits, 8 decimal places (±180.12345678)
- Provides GPS-level precision for mapping
SingleSelect with Colors:
- Support Level: Color-coded options for visual feedback
- Sign Size: Consistent option selection
- Category: Organized classification system

Backward Compatibility

The script maintains backward compatibility while using proper column types. Existing data migration may be needed if upgrading from the old schema.

Generated: July 5, 2025 Script Version: Column Type Optimized

12 KiB Raw Blame History