Hive Weight Analysis System - Detailed Manual. Eduardo Marcos Alejos. 04/08/2024

1. Introduction

The Hive Weight Analysis System is a sophisticated web application designed to visualize and analyze complex data related to the weight of beehives. This tool allows researchers and beekeepers to examine trends, patterns, and anomalies in hive weight data over time and across different geographical locations.

2. Architecture of the System

The Hive Weight Analysis System follows a single-page application (SPA) architecture with client-side processing. This architecture was chosen to provide a smooth and responsive user experience, minimizing server load.

3. Technologies Used

HTML5: Provides the basic structure of the application.
CSS3: Defines the styles and responsive design of the application.
JavaScript (ES6+): Implements all the application logic.
Chart.js (v3.7.0): Library for creating interactive and responsive charts.
Leaflet.js (v1.7.1): Library for creating interactive maps.
PapaParse (v5.3.0): Library for parsing CSV files.
Web Workers API: Allows background data processing.
Hammer.js: Library for handling touch events.
date-fns: Library for date manipulation.

4. Structure of the Code

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Hive Weight Analysis</title>
    <!-- Links to external libraries -->
</head>
<style>
    :root {
        /* CSS Variables */
    }
    /* Styles for components and layout */
</style>
<body>
    <div id="loadingOverlay" class="loading-overlay">
        <!-- Loading indicator -->
    </div>
    <div class="main-container">
        <div class="sidebar">
            <!-- Filtering and selection controls -->
        </div>
        <div class="content">
            <!-- Containers for visualizations -->
        </div>
    </div>
</body>
<script>
    // Global variables
    // Initialization functions
    // Data handling functions
    // UI update functions
    // Visualization creation and update functions
    // Event listeners and interactivity logic
</script>
</html>

5. Data Flow

The data flow in the application follows a unidirectional pattern, making it easy to track and debug. The data lifecycle is detailed as follows:

Initial Data Load

async function loadData() {
    document.getElementById('loadingOverlay').style.display = 'flex';
    const response = await fetch('euph_000027_scalesi_table_hivescale_raw.csv');
    const csvData = await response.text();
    const parsedData = Papa.parse(csvData, { header: true, dynamicTyping: true }).data;
    worker.postMessage({ action: 'processData', data: parsedData });
}

Data Processing in Web Worker

const worker = new Worker(URL.createObjectURL(new Blob([`
self.onmessage = function(e) {
    const { action, data } = e.data;
    if (action === 'processData') {
        const processedData = processData(data);
        self.postMessage({ action: 'processedData', data: processedData });
    }
};

function processData(data) {
    return data.map(row => ({
        ...row,
        yield: parseFloat(row.yield) || 0,
        positive_weight: Math.max(0, parseFloat(row.yield) || 0),
        daily_weight_change: parseFloat(row.yield) || 0,
        date: new Date(row.date),
        temperature: parseFloat(String(row.temperature).replace('.', '')) / 1000 || null,
        humidity: parseFloat(String(row.humidity).replace('.', '')) / 1000 || null
    }));
}
`], { type: 'text/javascript' })));

Receiving Processed Data

worker.onmessage = function(e) {
    const { action, data } = e.data;
    if (action === 'processedData') {
        allData = data;
        initializeFilters();
        initMap().then(() => {
            updateFilterTable();
            document.getElementById('loadingOverlay').style.display = 'none';
        });
    }
};

6. Main Components

6.1. Data Loading and Processing

Data loading and processing is a critical component of the application, as it sets the foundation for all subsequent operations.

Data Loading

async function loadData() {
    document.getElementById('loadingOverlay').style.display = 'flex';
    const response = await fetch('euph_000027_scalesi_table_hivescale_raw.csv');
    const csvData = await response.text();
    const parsedData = Papa.parse(csvData, { header: true, dynamicTyping: true }).data;
    worker.postMessage({ action: 'processData', data: parsedData });
}

Data Processing in Web Worker

const worker = new Worker(URL.createObjectURL(new Blob([`
self.onmessage = function(e) {
    const { action, data } = e.data;
    if (action === 'processData') {
        const processedData = processData(data);
        self.postMessage({ action: 'processedData', data: processedData });
    }
};

function processData(data) {
    return data.map(row => ({
        ...row,
        yield: parseFloat(row.yield) || 0,
        positive_weight: Math.max(0, parseFloat(row.yield) || 0),
        daily_weight_change: parseFloat(row.yield) || 0,
        date: new Date(row.date),
        temperature: parseFloat(String(row.temperature).replace('.', '')) / 1000 || null,
        humidity: parseFloat(String(row.humidity).replace('.', '')) / 1000 || null
    }));
}
`], { type: 'text/javascript' })));

Receiving Processed Data

worker.onmessage = function(e) {
    const { action, data } = e.data;
    if (action === 'processedData') {
        allData = data;
        initializeFilters();
        initMap().then(() => {
            updateFilterTable();
            document.getElementById('loadingOverlay').style.display = 'none';
        });
    }
};

6.2. Filtering System

The filtering system is a crucial part of the application, allowing users to refine the data displayed according to specific criteria.

Filter Initialization

function initializeFilters() {
    const years = [...new Set(allData.map(row => row.date.getFullYear()))];
    const yearSelect = document.getElementById('yearSelect');
    const secondYearSelect = document.getElementById('secondYearSelect');
    years.forEach(year => {
        const option = document.createElement('option');
        option.value = year;
        option.textContent = year;
        yearSelect.appendChild(option);

        const secondOption = option.cloneNode(true);
        secondYearSelect.appendChild(secondOption);
    });
    yearSelect.value = "2014";

    // ... (initialization of other filters)

    initializeTableFilters();
}

Table Filters

function initializeTableFilters() {
    const columns = ['region', 'locality', 'area'];
    columns.forEach(column => {
        const select = document.getElementById(`${column}Filter`);
        const input = document.getElementById(`${column}FilterInput`);
        const autocompleteDiv = document.getElementById(`${column}Autocomplete`);

        // ... (styles and position configuration)

        input.addEventListener('input', debounce(() => {
            const value = input.value.toLowerCase();
            const filteredValues = values.filter(v => v.toLowerCase().includes(value));
            showAutocompleteSuggestions(filteredValues, autocompleteDiv, input, select, column);
            updateSelectOptions(filteredValues, select);
            currentFilters[column] = value;
            updateFilterTable();
        }, 300));

        // ... (event handling for selection and suggestion closure)
    });
}

Updating Dependent Filters

function updateDependentFilters(filteredData) {
    ['region', 'locality', 'area'].forEach(field => {
        const select = document.getElementById(`${field}Filter`);
        const input = document.getElementById(`${field}FilterInput`);
        const uniqueValues = [...new Set(filteredData.map(row => row[field]))].filter(Boolean).sort();
        select.innerHTML = '';
        uniqueValues.forEach(value => {
            const option = document.createElement('option');
            option.value = value;
            option.textContent = value;
            select.appendChild(option);
        });
        if (currentFilters[field] !== 'All') {
            select.value = currentFilters[field];
            input.value = currentFilters[field];
        }
    });
}

6.3. Visualizations

6.3.1. Temporal Line Chart

function createChart(data, minMax) {
    const ctx = document.getElementById('myChart').getContext('2d');
    return new Chart(ctx, {
        type: 'line',
        data: { datasets: data },
        options: {
            responsive: true,
            maintainAspectRatio: false,
            interaction: { mode: 'nearest', axis: 'x', intersect: false },
            scales: {
                x: { 
                    type: 'time', 
                    time: { 
                        unit: 'month',
                        displayFormats: {
                            month: 'MMM yyyy'
                        }
                    },
                    title: { display: true, text: 'Date' },
                    // ... (additional axis configuration)
                },
                y: { 
                    title: { display: true, text: document.getElementById('measureSelect').value },
                    suggestedMin: minMax.min,
                    suggestedMax: minMax.max
                }
            },
            plugins: {
                tooltip: {
                    callbacks: {
                        label: function(context) {
                            const dataPoint = context.raw;
                            return [
                                `${context.dataset.label}`,
                                `Weight: ${dataPoint.y.toFixed(2)}`,
                                `Date: ${dataPoint.x.toLocaleDateString()}`,
                                `ScaleID: ${dataPoint.scaleID}`,
                                `Region: ${dataPoint.region}`
                            ];
                        }
                    }
                },
                zoom: {
                    zoom: {
                        wheel: { enabled: true },
                        pinch: { enabled: true },
                        mode: 'xy',
                    },
                    pan: {
                        enabled: true,
                        mode: 'xy',
                    }
                }
            },
            // ... (additional configuration)
        }
    });
}

6.3.2. Interactive Map

async function initMap() {
    map = L.map('map').setView([46.1512, 14.9955], 7);
    L.tileLayer('https://{s}.basemaps.cartocdn.com/rastertiles/voyager/{z}/{x}/{y}{r}.png', {
        attribution: '© OpenStreetMap contributors © CARTO'
    }).addTo(map);

    fetch('si(simple).json')
        .then(response => response.json())
        .then(geoJsonData => {
            L.geoJSON(geoJsonData, {
                style: function (feature) {
                    return { color: "#000", weight: 1, fillOpacity: 0 };
                }
            }).addTo(map);
        })
        .catch(error => console.error('Error loading GeoJSON:', error));

    markerClusterGroup = L.markerClusterGroup({
        maxClusterRadius: 30,
        spiderfyOnMaxZoom: false
    });
    map.addLayer(markerClusterGroup);
}

function updateMap(filteredData) {
    if (!map || !markerClusterGroup) {
        console.error('Map or markerClusterGroup not initialized');
        return;
    }

    markerClusterGroup.clearLayers();

    const localityData = filteredData.reduce((acc, row) => {
        // ... (data aggregation by locality)
    }, {});

    Object.entries(localityData).forEach(([locality, data]) => {
        if (!isNaN(data.lat) && !isNaN(data.lon) && data.lat !== 0 && data.lon !== 0) {
            const marker = L.circleMarker([data.lat, data.lon], {
                radius: 8,
                fillColor: getColor(data.maxWeight),
                color: "#000",
                weight: 1,
                opacity: 1,
                fillOpacity: 0.8
            });

            marker.bindPopup(`
                Locality: ${locality}

                Max Weight: ${data.maxWeight.toFixed(1)} kg

                ScaleID: ${data.scaleID}

                Date: ${data.date instanceof Date ? data.date.toISOString().split('T')[0] : data.date}

                NUTS3: ${data.region || 'N/A'}

                Measurements: ${data.count}
            `);

            markerClusterGroup.addLayer(marker);
        }
    });

    // ... (adjust map view)
}

6.3.3. Data Table

function updateFilterTable(yearChanged = false) {
    const tableBody = document.getElementById('filterTableBody');
    tableBody.innerHTML = '';
    const selectedYear = parseInt(document.getElementById('yearSelect').value);
    const selectedMeasure = document.getElementById('measureSelect').value;
    const secondYear = document.getElementById('secondYearSelect').value;

    let filteredData = allData.filter(row => row.date.getFullYear() === selectedYear);
    let secondFilteredData = secondYear !== 'none' ? allData.filter(row => row.date.getFullYear() === parseInt(secondYear)) : [];

    // ... (additional filters application)

    const fragment = document.createDocumentFragment();
    filteredData.slice(0, 100).forEach(row => {
        const tr = document.createElement('tr');
        let measureValue;
        switch (selectedMeasure) {
            case 'yield':
                measureValue = row.yield;
                break;
            case 'positive_weight':
                measureValue = row.positive_weight;
                break;
            case 'daily_weight_change':
                measureValue = row.daily_weight_change;
                break;
        }
        tr.innerHTML = `
            ${row.id || ''}
            ${typeof measureValue === 'number' ? measureValue.toFixed(1) : measureValue || ''}
            ${row.date.toISOString().split('T')[0] || ''}
            ${row.region || ''}
            ${row.locality || ''}
            ${row.area || ''}
        `;
        fragment.appendChild(tr);
    });
    tableBody.appendChild(fragment);

    // ... (update dependent filters and visualizations)
}

6.3.4. Scatter Plot

function updateScatterChart(filteredData) {
    const ctx = document.getElementById('scatterChart').getContext('2d');

    if (scatterChart) {
        scatterChart.destroy();
    }

    const data = filteredData
        .filter(row => row.temperature !== null && row.humidity !== null && !isNaN(row.temperature) && !isNaN(row.humidity))
        .map(row => ({
            x: row.temperature,
            y: row.humidity,
            r: Math.abs(row.yield) * 5 + 2,
            yield: row.yield
        }));

    if (data.length === 0) {
        ctx.font = '20px Arial';
        ctx.fillStyle = 'black';
        ctx.textAlign = 'center';
        ctx.fillText('No data available for this visualization', ctx.canvas.width / 2, ctx.canvas.height / 2);
        return;
    }

    scatterChart = new Chart(ctx, {
        type: 'bubble',
        data: {
            datasets: [{
                label: 'Humidity vs Temperature',
                data: data,
                backgroundColor: data.map(d => `rgba(255, 99, 132, ${Math.min(Math.abs(d.yield) / 2 + 0.1, 1)})`)
            }]
        },
        options: {
            // ... (chart configuration options)
        }
    });
}

7. User Interactivity and Experience

Dynamic Filters: Filters update in real-time as the user types, with autocomplete to facilitate value selection.
Zoom and Pan on Charts: Users can zoom and pan on charts to explore data in detail.
Informative Tooltips: All charts and the map include detailed tooltips on hover.
Dynamic View Switching: Users can switch between different visualizations (chart, map, table, scatter) without losing filter context.
Loading Indicator: An overlay is displayed during long operations, such as the initial data load.
Responsive Design: The interface adapts to different screen sizes for a consistent experience on mobile and desktop devices.

8. Optimization and Performance

Web Worker: Initial data processing is done in a Web Worker to avoid blocking the UI thread.
Debounce on Filters: Debounce is used on text filters to reduce the frequency of updates during typing.
DocumentFragment: A DocumentFragment is used to minimize DOM manipulation when updating the data table.
Limited Data Display: The data table is limited to displaying 100 rows to maintain rendering performance.
Marker Clustering on Map: Leaflet.markercluster is used to cluster nearby markers, significantly reducing the number of rendered elements on the map.
Asynchronous Resource Loading: Data and external resources are loaded asynchronously to avoid blocking the initial page load.
requestAnimationFrame for Animations: requestAnimationFrame is used for smooth animations, especially when updating charts.
Memoization of Expensive Calculations: Memoization is implemented for frequently repeated calculations with the same parameters.
CSS will-change: The CSS will-change property is used for elements that are frequently animated to optimize rendering performance.

9. Error Handling and Edge Cases

Robust error handling and edge case management are crucial for the stability and usability of the application:

Data Loading

async function loadData() {
    try {
        document.getElementById('loadingOverlay').style.display = 'flex';
        const response = await fetch('euph_000027_scalesi_table_hivescale_raw.csv');
        if (!response.ok) {
            throw new Error(`HTTP error! status: ${response.status}`);
        }
        const csvData = await response.text();
        // Data processing...
    } catch (error) {
        console.error('Error loading data:', error);
        showErrorMessage('Failed to load data. Please try again later.');
    } finally {
        document.getElementById('loadingOverlay').style.display = 'none';
    }
}

Data Validation

function validateDataPoint(point) {
    return (
        isFinite(point.yield) &&
        point.date instanceof Date &&
        !isNaN(point.date.getTime()) &&
        typeof point.region === 'string' &&
        typeof point.locality === 'string'
    );
}

Handling Missing Data

function getDataPointValue(point, measure) {
    switch (measure) {
        case 'yield':
            return isFinite(point.yield) ? point.yield : null;
        case 'positive_weight':
            return isFinite(point.yield) ? Math.max(0, point.yield) : null;
        // ... other cases
        default:
            return null;
    }
}

Limits on Filters and Visualizations

const MAX_DISPLAYED_POINTS = 10000;
const MAX_TABLE_ROWS = 100;

Error Handling in Web Worker

worker.onerror = function (error) {
    console.error('Web Worker error:', error);
    showErrorMessage('An error occurred while processing data. Please refresh the page.');
};

10. Customization and Extensibility

The system is designed to be easily customizable and extensible:

Color and Style Configuration

:root {
    --primary-color: #3498db;
    --secondary-color: #2ecc71;
    --background-color: #ecf0f1;
    --text-color: #34495e;
    --border-color: #bdc3c7;
}

Chart Configuration

const chartOptions = {
    responsive: true,
    maintainAspectRatio: false,
    // ... other options
};

Extending Filters

function addNewFilter(filterName, filterOptions) {
    // Logic to add a new filter
}

Adding New Visualizations

function addNewVisualization(name, renderFunction) {
    visualizations[name] = renderFunction;
    // Update UI to include the new visualization
}

API for Plugins

const pluginAPI = {
    addFilter: addNewFilter,
    addVisualization: addNewVisualization,
    getData: () => allData,
    // ... other useful methods
};

11. Programming Best Practices

Modularity: Code is organized into small, specific functions, each with a clear responsibility.
Descriptive Names: Variables and functions have descriptive names that clearly indicate their purpose.
Comments and Documentation: Comments are included to explain complex or non-obvious logic. Each main function has a comment describing its purpose, parameters, and return value.
Consistency in Code Style: Consistent code style is maintained throughout the project (indentation, naming, etc.).
Avoid Code Duplication: Refactoring is used to eliminate duplicated code and improve maintainability.
State Management: Application state is managed centrally to avoid inconsistencies.
Use of Constants: Constants are defined for values used in multiple places:

const DEFAULT_YEAR = 2014;
const MAX_CHART_POINTS = 1000;

Input Validation: User inputs and data are validated before being processed.
Avoid Global Variables: The use of global variables is minimized, preferring parameter passing and closures.
Handling Promises: async/await is used for cleaner and easier-to-read asynchronous operations code.

12. Security Considerations

Data Sanitization: All user-entered data is sanitized before use:

function sanitizeInput(input) {
    return input.replace(/[&<>"']/g, function (m) {
        return {
            '&': '&',
            '<': '<',
            '>': '>',
            '"': '"',
            "'": '''
        }[m];
    });
}

XSS Prevention: textContent is used instead of innerHTML whenever possible to prevent XSS attacks.
Use of HTTPS: It is recommended to serve the application over HTTPS to protect the integrity and confidentiality of the data.
Secure Handling of Sensitive Data: Sensitive data is not stored in the local storage of the browser.
Server-Side Validation: Although the application is primarily client-side, it is recommended to implement validations on the server for any APIs used.