project overview algolia

Enhancing Our Search Experience with Algolia Integration

Questions and Next Steps

Collaboration with Algolia Team

Proposed Solutions and Action Plan

Prioritized Goals and Features

Current State of Integration

Project Overview and Objectives

We are a eccomerce firearms retailer prioritizing high volume/turnover.We have a wide variety of products from firearms to apparell, accessories, etc..We have a high number of products (92,500-100,000) with frequent updates and new products (daily)Coreware is our hosting platform, there is currently a robust backend of data including thorough taxonomy and facets.Several integrations for importing products managing the site etc..Is semi built for/currently using csv's for data transfers for the most partEverything said to be taken with a grain of salt and am very open to different solutions/solves but this is the current state

Company Background - Battlehawk Armory

Key Objectives:Enhance search functionality on the website.Improve user experience through personalization.Automate data synchronization between Coreware and Algolia.Implement robust analytics and tracking.Ensure scalability, security, and data protection.

Project Goals

API Connection

Description of the hand-built API for indexing.Current use of Coreware API for product data.Overview of the existing autocomplete front end.Mention of facets already set up and retrievable via API.

Existing Infrastructure

Obviously there is a large number of api funcitons, aswell as there being some custom made ones but I beleive this likely contains much of the necessary information for indexing. notably it also contains a time changed and date created allowing for a potential automated update solve

" "product_addons": [] }, "api_domain_name": "battlehawkarmory.coreware.com", "result": "OK"

" "minimum_price": "", "manufacturer_advertised_price": "", "map_expiration_date": "", "width": "8.0000", "length": "10.0000", "height": "2.0000", "weight": "4.0000", "product_manufacturer_code": "HECKLER_KOCH_INC.", "product_manufacturer_description": "Heckler & Koch, Inc.", "product_category_codes": "PISTOLS", "product_tag_codes": "FFL_REQUIRED", "product_facets": "ACTION||Single/Double||||BARREL_LENGTH||3.58\"||||BARREL_LENGTH_RANGE||3.00\" to 3.99\"||||BARREL_TYPE||Polygonal Rifled||||CALIBER||.40 S&W||||CAPACITY||10+1||||FRAME_FINISH||Black||||FRAME_MATERIAL||Polymer||||GRIPS||Black Polymer Grip/Frame||||INCLUDES||3 Magazines||||OAL||6.81\"||||SAFETY||Decocker||||SEARCH_CODE||HK||||SERIES||V1 *CA/MA Compliant*||||SIGHT_CONFIGURATION||Night||||SIGHT_STYLE||Night||||SIMILAR_ITEMS||HG40||||SLIDE_DESCRIPTION||Blued Steel||||TYPE||Pistol||||WEIGHT||27.52 oz||||WEIGHT_RANGE||16.00 oz to 31.99 oz", "related_products": "642230254312||LEGAL_PROTECTION", "restricted_states": "", "alternate_images": "", "product_prices": [], "product_videos": [], "product_custom_fields": [], "custom_fields": [], "product_addons": [] }, "api_domain_name": "battlehawkarmory.coreware.com", "result": "OK"

" "search_multiplier": "1.00", "calculated_search_multiplier": "1.00", "points_multiplier": "1.00", "sort_order": 100, "requires_user": 0, "catalog_lock": 1, "no_update": 0, "not_searchable": 0, "cannot_dropship": 0, "reindex": 0, "custom_product": 0, "non_inventory_item": 0, "not_taxable": 0, "serializable": 1, "no_online_order": 0, "in_stock": 0, "internal_use_only": 0, "inactive": 0, "notes": "", "version": 1, "product_data_id": 196102, "product_distributor_id": 1, "unit_id": 44, "model": "USP40C", "upc_code": "642230254312", "isbn": "", "isbn_13": "", "manufacturer_sku": "704001LELA5", "minimum_price": "", "manufacturer_advertised_price": "", "map_expiration_date": "", "width": "8.0000", "length": "10.0000", "height": "2.0000", "weight": "4.0000", "product_manufacturer_code": "HECKLER_KOCH_INC.", "product_manufacturer_description": "Heckler & Koch, Inc.", "product_category_codes": "PISTOLS", "product_tag_codes": "FFL_REQUIRED", "product_facets": "ACTION||Single/Double||||BARREL_LENGTH||3.58\"||||BARREL_LENGTH_RANGE||3.00\" to 3.99\"||||BARREL_TYPE||Polygonal Rifled||||CALIBER||.40 S&W||||CAPACITY||10+1||||FRAME_FINISH||Black||||FRAME_MATERIAL||Polymer||||GRIPS||Black Polymer Grip/Frame||||INCLUDES||3 Magazines||||OAL||6.81\"||||SAFETY||Decocker||||SEARCH_CODE||HK||||SERIES||V1 *CA/MA Compliant*||||SIGHT_CONFIGURATION||Night||||SIGHT_STYLE||Night||||SIMILAR_ITEMS||HG40||||SLIDE_DESCRIPTION||Blued Steel||||TYPE||Pistol||||WEIGHT||27.52 oz||||WEIGHT_RANGE||16.00 oz to 31.99 oz", "related_products": "642230254312||LEGAL_PROTECTION", "restricted_states": "", "alternate_images": "", "product_prices": [], "product_videos": [], "product_custom_fields": [], "custom_fields": [], "product_addons": [] }, "api_domain_name": "battlehawkarmory.coreware.com", "result": "OK"

" "link_name": "hk-704001lela5-usp40c-cama-compliant-40-sw-3.58-101-black-blued-steel-black-polymer-grip", "product_type_id": "", "product_format_id": "", "product_manufacturer_id": 8578, "base_cost": "816.90", "list_price": "", "pricing_structure_id": "", "low_inventory_quantity": 0, "low_inventory_surcharge_amount": "", "virtual_product": 0, "file_id": "", "image_id": "", "remote_identifier": 2400, "cart_minimum": "", "cart_maximum": "", "order_maximum": "", "date_created": "2019-12-03", "expiration_date": "", "time_changed": "2023-04-04 19:18:18", "user_group_id": "", "error_message": "", "search_terms": "", "tax_rate_id": "", "search_multiplier": "1.00", "calculated_search_multiplier": "1.00", "points_multiplier": "1.00", "sort_order": 100, "requires_user": 0, "catalog_lock": 1, "no_update": 0, "not_searchable": 0, "cannot_dropship": 0, "reindex": 0, "custom_product": 0, "non_inventory_item": 0, "not_taxable": 0, "serializable": 1, "no_online_order": 0, "in_stock": 0, "internal_use_only": 0, "inactive": 0, "notes": "", "version": 1, "product_data_id": 196102, "product_distributor_id": 1, "unit_id": 44, "model": "USP40C", "upc_code": "642230254312", "isbn": "", "isbn_13": "", "manufacturer_sku": "704001LELA5", "minimum_price": "", "manufacturer_advertised_price": "", "map_expiration_date": "", "width": "8.0000", "length": "10.0000", "height": "2.0000", "weight": "4.0000", "product_manufacturer_code": "HECKLER_KOCH_INC.", "product_manufacturer_description": "Heckler & Koch, Inc.", "product_category_codes": "PISTOLS", "product_tag_codes": "FFL_REQUIRED", "product_facets": "ACTION||Single/Double||||BARREL_LENGTH||3.58\"||||BARREL_LENGTH_RANGE||3.00\" to 3.99\"||||BARREL_TYPE||Polygonal Rifled||||CALIBER||.40 S&W||||CAPACITY||10+1||||FRAME_FINISH||Black||||FRAME_MATERIAL||Polymer||||GRIPS||Black Polymer Grip/Frame||||INCLUDES||3 Magazines||||OAL||6.81\"||||SAFETY||Decocker||||SEARCH_CODE||HK||||SERIES||V1 *CA/MA Compliant*||||SIGHT_CONFIGURATION||Night||||SIGHT_STYLE||Night||||SIMILAR_ITEMS||HG40||||SLIDE_DESCRIPTION||Blued Steel||||TYPE||Pistol||||WEIGHT||27.52 oz||||WEIGHT_RANGE||16.00 oz to 31.99 oz", "related_products": "642230254312||LEGAL_PROTECTION", "restricted_states": "", "alternate_images": "", "product_prices": [], "product_videos": [], "product_custom_fields": [], "custom_fields": [], "product_addons": [] }, "api_domain_name": "battlehawkarmory.coreware.com", "result": "OK"

"post_variables": { "method": "get_full_product_information\n", "connection_key": "ope", "product_id": "196103", "upc_code": "SIGE26R9LEGIONSAORXP", "action": "get_full_product_information\n" }, "product_information": { "product_id": 196103, "client_id": 5, "product_code": "7079", "description": "HK 704001LELA5 USP40C *CA/MA Compliant 40 S&W 3.58\" 10+1 Black Blued Steel Black Polymer Grip", "detailed_description": "The HK USP (Universal Self-loading Pistol) is an accurate and ultra-reliable handgun. It utilizes a fiber-reinforced polymer frame stiffened by stainless steel inserts at areas subject to stress and friction. The control lever, a combination safety and decocking lever, is frame mounted and quickly accessible, unlike the slide-mounted safeties. The modular design of the USP's internal components allows the control lever function to be switched from the left to the right side of the pistol for left-handed shooters. The USP Compact are smaller and lighter than large frame USPs. The reduction in trigger reach and grip circumference increases concealability and enhances shooting ergonomics. It utilizes a narrow, full-hand grip frame with a choice of interchangeable extended or flush-fitting magazine floorplates. The USP Compact features a polygonal bore barrel, one-piece steel slide with contoured, extended release, bobbed hammer, universal mounting grooves, and ambidextrous magazine release lever. This model has a black frame with blued slide, captive recoil spring assembly with polymer absorber bushing, night sights, and comes with three 10-round magazines.", "link_name": "hk-704001lela5-usp40c-cama-compliant-40-sw-3.58-101-black-blued-steel-black-polymer-grip", "product_type_id": "", "product_format_id": "", "product_manufacturer_id": 8578, "base_cost": "816.90", "list_price": "", "pricing_structure_id": "", "low_inventory_quantity": 0, "low_inventory_surcharge_amount": "", "virtual_product": 0, "file_id": "", "image_id": "", "remote_identifier": 2400, "cart_minimum": "", "cart_maximum": "", "order_maximum": "", "date_created": "2019-12-03", "expiration_date": "", "time_changed": "2023-04-04 19:18:18", "user_group_id": "", "error_message": "", "search_terms": "", "tax_rate_id": "", "search_multiplier": "1.00", "calculated_search_multiplier": "1.00", "points_multiplier": "1.00", "sort_order": 100, "requires_user": 0, "catalog_lock": 1, "no_update": 0, "not_searchable": 0, "cannot_dropship": 0, "reindex": 0, "custom_product": 0, "non_inventory_item": 0, "not_taxable": 0, "serializable": 1, "no_online_order": 0, "in_stock": 0, "internal_use_only": 0, "inactive": 0, "notes": "", "version": 1, "product_data_id": 196102, "product_distributor_id": 1, "unit_id": 44, "model": "USP40C", "upc_code": "642230254312", "isbn": "", "isbn_13": "", "manufacturer_sku": "704001LELA5", "minimum_price": "", "manufacturer_advertised_price": "", "map_expiration_date": "", "width": "8.0000", "length": "10.0000", "height": "2.0000", "weight": "4.0000", "product_manufacturer_code": "HECKLER_KOCH_INC.", "product_manufacturer_description": "Heckler & Koch, Inc.", "product_category_codes": "PISTOLS", "product_tag_codes": "FFL_REQUIRED", "product_facets": "ACTION||Single/Double||||BARREL_LENGTH||3.58\"||||BARREL_LENGTH_RANGE||3.00\" to 3.99\"||||BARREL_TYPE||Polygonal Rifled||||CALIBER||.40 S&W||||CAPACITY||10+1||||FRAME_FINISH||Black||||FRAME_MATERIAL||Polymer||||GRIPS||Black Polymer Grip/Frame||||INCLUDES||3 Magazines||||OAL||6.81\"||||SAFETY||Decocker||||SEARCH_CODE||HK||||SERIES||V1 *CA/MA Compliant*||||SIGHT_CONFIGURATION||Night||||SIGHT_STYLE||Night||||SIMILAR_ITEMS||HG40||||SLIDE_DESCRIPTION||Blued Steel||||TYPE||Pistol||||WEIGHT||27.52 oz||||WEIGHT_RANGE||16.00 oz to 31.99 oz", "related_products": "642230254312||LEGAL_PROTECTION", "restricted_states": "", "alternate_images": "", "product_prices": [], "product_videos": [], "product_custom_fields": [], "custom_fields": [], "product_addons": [] }, "api_domain_name": "battlehawkarmory.coreware.com", "result": "OK"

Get_full_product info api call return data

// Function to index product data to Algoliaasync function indexProductData(productData) { try { // Function to parse product_facets string into an object function parseFacets(facetsString) { const facetsArray = facetsString.split('||||'); const facetsObject = {}; for (let i = 0; i < facetsArray.length; i += 2) { const key = facetsArray[i]; const value = facetsArray[i + 1]; if (key && value) { facetsObject[key] = value; } } return facetsObject; } // Fetch the image URL using the product's slug const imageUrl = await fetchImageUrl(productData.link_name); const algoliaProduct = { objectID: productData.product_id, link_name: productData.link_name, description: productData.description, product_code: productData.product_code, model: productData.model, upc_code: productData.upc_code, category: productData.product_category_codes, tags: productData.product_tag_codes, image_url: imageUrl || productData.image_url || null, // New fields search_terms: productData.search_terms, product_manufacturer_description: productData.product_manufacturer_description, product_facets: parseFacets(productData.product_facets), text_data: productData.text_data, in_stock: productData.in_stock, time_changed: productData.time_changed, }; await index.saveObject(algoliaProduct); console.log( `Product ID ${productData.product_id} indexed successfully to Algolia!` ); } catch (error) { console.error( `Error indexing product ID ${productData.product_id}:`, error.response ? error.response.data : error.message ); throw error; } } // Main function to fetch and push product data to Algolia async function fetchAndPushToAlgolia() { try { const productIds = [20]; // Replace with dynamic IDs or implement logic to fetch multiple IDs for (const productId of productIds) { const productData = await fetchProductData(productId); await indexProductData(productData); } } catch (error) { console.error('Error in fetchAndPushToAlgolia:', error.message); } } // Run the main function fetchAndPushToAlgolia();

const response = await axios.get(url, { headers: { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)', Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'Accept-Language': 'en-US,en;q=0.9', }, }); const html = response.data; const $ = cheerio.load(html); // Adjust the selector based on your HTML structure const imageSrc = $('#_product_primary_image .product-image').attr('src') || null; return imageSrc; } catch (error) { console.error('Error fetching image URL:', error.message); return null; } } // Function to index product data to Algolia async function indexProductData(productData) { try { // Function to parse product_facets string into an object function parseFacets(facetsString) { const facetsArray = facetsString.split('||||'); const facetsObject = {}; for (let i = 0; i < facetsArray.length; i += 2) { const key = facetsArray[i]; const value = facetsArray[i + 1]; if (key && value) { facetsObject[key] = value; } } return facetsObject; } // Fetch the image URL using the product's slug const imageUrl = await fetchImageUrl(productData.link_name); const algoliaProduct = { objectID: productData.product_id, link_name: productData.link_name, description: productData.description, product_code: productData.product_code, model: productData.model, upc_code: productData.upc_code, category: productData.product_category_codes, tags: productData.product_tag_codes, image_url: imageUrl || productData.image_url || null, // New fields search_terms: productData.search_terms, product_manufacturer_description: productData.product_manufacturer_description, product_facets: parseFacets(productData.product_facets), text_data: productData.text_data, in_stock: productData.in_stock, time_changed: productData.time_changed, }; await index.saveObject(algoliaProduct); console.log( `Product ID ${productData.product_id} indexed successfully to Algolia!` ); } catch (error) { console.error( `Error indexing product ID ${productData.product_id}:`, error.response ? error.response.data : error.message ); throw error; } } // Main function to fetch and push product data to Algolia async function fetchAndPushToAlgolia() { try { const productIds = [20]; // Replace with dynamic IDs or implement logic to fetch multiple IDs for (const productId of productIds) { const productData = await fetchProductData(productId); await indexProductData(productData); } } catch (error) { console.error('Error in fetchAndPushToAlgolia:', error.message); } } // Run the main function fetchAndPushToAlgolia();

method: 'get_full_product_information', product_id: productId, connection_key: connectionKey, }, }); if (response.status !== 200 || !response.data.product_information) { throw new Error(`Failed to fetch product data for ID ${productId}`); } return response.data.product_information; } catch (error) { console.error( `Error fetching product data for ID ${productId}:`, error.response ? error.response.data : error.message ); throw error; } } // Function to fetch image URL using Axios and Cheerio async function fetchImageUrl(slug) { try { const url = `https://battlehawkarmory.com/product/${slug}`; const response = await axios.get(url, { headers: { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)', Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'Accept-Language': 'en-US,en;q=0.9', }, }); const html = response.data; const $ = cheerio.load(html); // Adjust the selector based on your HTML structure const imageSrc = $('#_product_primary_image .product-image').attr('src') || null; return imageSrc; } catch (error) { console.error('Error fetching image URL:', error.message); return null; } } // Function to index product data to Algolia async function indexProductData(productData) { try { // Function to parse product_facets string into an object function parseFacets(facetsString) { const facetsArray = facetsString.split('||||'); const facetsObject = {}; for (let i = 0; i < facetsArray.length; i += 2) { const key = facetsArray[i]; const value = facetsArray[i + 1]; if (key && value) { facetsObject[key] = value; } } return facetsObject; } // Fetch the image URL using the product's slug const imageUrl = await fetchImageUrl(productData.link_name); const algoliaProduct = { objectID: productData.product_id, link_name: productData.link_name, description: productData.description, product_code: productData.product_code, model: productData.model, upc_code: productData.upc_code, category: productData.product_category_codes, tags: productData.product_tag_codes, image_url: imageUrl || productData.image_url || null, // New fields search_terms: productData.search_terms, product_manufacturer_description: productData.product_manufacturer_description, product_facets: parseFacets(productData.product_facets), text_data: productData.text_data, in_stock: productData.in_stock, time_changed: productData.time_changed, }; await index.saveObject(algoliaProduct); console.log( `Product ID ${productData.product_id} indexed successfully to Algolia!` ); } catch (error) { console.error( `Error indexing product ID ${productData.product_id}:`, error.response ? error.response.data : error.message ); throw error; } } // Main function to fetch and push product data to Algolia async function fetchAndPushToAlgolia() { try { const productIds = [20]; // Replace with dynamic IDs or implement logic to fetch multiple IDs for (const productId of productIds) { const productData = await fetchProductData(productId); await indexProductData(productData); } } catch (error) { console.error('Error in fetchAndPushToAlgolia:', error.message); } } // Run the main function fetchAndPushToAlgolia();

require('dotenv').config(); const algoliasearch = require('algoliasearch'); const axios = require('axios'); const cheerio = require('cheerio'); // Initialize Algolia client using credentials from environment variables const client = algoliasearch( process.env.ALGOLIA_APP_ID, process.env.ALGOLIA_ADMIN_API_KEY ); const index = client.initIndex(process.env.ALGOLIA_INDEX_NAME); // Coreware API endpoint and credentials const corewareURL = process.env.COREWARE_API_URL; const connectionKey = process.env.COREWARE_CONNECTION_KEY; // Function to fetch product data from Coreware API async function fetchProductData(productId) { try { const response = await axios.get(corewareURL, { params: { method: 'get_full_product_information', product_id: productId, connection_key: connectionKey, }, }); if (response.status !== 200 || !response.data.product_information) { throw new Error(`Failed to fetch product data for ID ${productId}`); } return response.data.product_information; } catch (error) { console.error( `Error fetching product data for ID ${productId}:`, error.response ? error.response.data : error.message ); throw error; } } // Function to fetch image URL using Axios and Cheerio async function fetchImageUrl(slug) { try { const url = `https://battlehawkarmory.com/product/${slug}`; const response = await axios.get(url, { headers: { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)', Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'Accept-Language': 'en-US,en;q=0.9', }, }); const html = response.data; const $ = cheerio.load(html); // Adjust the selector based on your HTML structure const imageSrc = $('#_product_primary_image .product-image').attr('src') || null; return imageSrc; } catch (error) { console.error('Error fetching image URL:', error.message); return null; } } // Function to index product data to Algolia async function indexProductData(productData) { try { // Function to parse product_facets string into an object function parseFacets(facetsString) { const facetsArray = facetsString.split('||||'); const facetsObject = {}; for (let i = 0; i < facetsArray.length; i += 2) { const key = facetsArray[i]; const value = facetsArray[i + 1]; if (key && value) { facetsObject[key] = value; } } return facetsObject; } // Fetch the image URL using the product's slug const imageUrl = await fetchImageUrl(productData.link_name); const algoliaProduct = { objectID: productData.product_id, link_name: productData.link_name, description: productData.description, product_code: productData.product_code, model: productData.model, upc_code: productData.upc_code, category: productData.product_category_codes, tags: productData.product_tag_codes, image_url: imageUrl || productData.image_url || null, // New fields search_terms: productData.search_terms, product_manufacturer_description: productData.product_manufacturer_description, product_facets: parseFacets(productData.product_facets), text_data: productData.text_data, in_stock: productData.in_stock, time_changed: productData.time_changed, }; await index.saveObject(algoliaProduct); console.log( `Product ID ${productData.product_id} indexed successfully to Algolia!` ); } catch (error) { console.error( `Error indexing product ID ${productData.product_id}:`, error.response ? error.response.data : error.message ); throw error; } } // Main function to fetch and push product data to Algolia async function fetchAndPushToAlgolia() { try { const productIds = [20]; // Replace with dynamic IDs or implement logic to fetch multiple IDs for (const productId of productIds) { const productData = await fetchProductData(productId); await indexProductData(productData); } } catch (error) { console.error('Error in fetchAndPushToAlgolia:', error.message); } } // Run the main function fetchAndPushToAlgolia();

custom api

The product is pulled via product code, we can input as many as needed, and could create logic for automating indexing, can also tweak which fields for pulling different info.

} catch (error) { console.error('Error in fetchAndPushToAlgolia:', error.message); } } // Run the main function fetchAndPushToAlgolia();

}; await index.saveObject(algoliaProduct); console.log( `Product ID ${productData.product_id} indexed successfully to Algolia!` ); } catch (error) { console.error( `Error indexing product ID ${productData.product_id}:`, error.response ? error.response.data : error.message ); throw error; } } // Main function to fetch and push product data to Algolia async function fetchAndPushToAlgolia() { try { const productIds = [20]; // Replace with dynamic IDs or implement logic to fetch multiple IDs for (const productId of productIds) { const productData = await fetchProductData(productId); await indexProductData(productData); } } catch (error) { console.error('Error in fetchAndPushToAlgolia:', error.message); } } // Run the main function fetchAndPushToAlgolia();

// Fetch the image URL using the product's slug const imageUrl = await fetchImageUrl(productData.link_name); const algoliaProduct = { objectID: productData.product_id, link_name: productData.link_name, description: productData.description, product_code: productData.product_code, model: productData.model, upc_code: productData.upc_code, category: productData.product_category_codes, tags: productData.product_tag_codes, image_url: imageUrl || productData.image_url || null, // New fields search_terms: productData.search_terms, product_manufacturer_description: productData.product_manufacturer_description, product_facets: parseFacets(productData.product_facets), text_data: productData.text_data, in_stock: productData.in_stock, time_changed: productData.time_changed, }; await index.saveObject(algoliaProduct); console.log( `Product ID ${productData.product_id} indexed successfully to Algolia!` ); } catch (error) { console.error( `Error indexing product ID ${productData.product_id}:`, error.response ? error.response.data : error.message ); throw error; } } // Main function to fetch and push product data to Algolia async function fetchAndPushToAlgolia() { try { const productIds = [20]; // Replace with dynamic IDs or implement logic to fetch multiple IDs for (const productId of productIds) { const productData = await fetchProductData(productId); await indexProductData(productData); } } catch (error) { console.error('Error in fetchAndPushToAlgolia:', error.message); } } // Run the main function fetchAndPushToAlgolia();

custom api cont.

All that results in this index. using the current headless browser system it takes about 3 seconds per product to index.Though when looking at updates we wont have to pull images, not having to use the headless browser drastically reducing that time to search through the catalog for date ranges in order to than reindex/update.

Front end looks like this currently, when clicking into the search box it opens the autocomplete, when clicking out it closes it.NOTE: when clicking the search icon it reverts to our old search functionality showing our product display results page.Integrated through java css and html.

its further down the priority list but once we have a solid configuration base, indexing etc.. adding a sort of menu highlighting top trending items, facets, etc. replacing this leftmost collumn could be good

Push product id's of updated products for reindexing

pull full product data-run time changed date range

Can maybe grab from here

API - - algolia index

catalog update

Manual product entry/updates

3rd party integrations via php script uploading new products backend

Data Flow Diagram

List of Challenges:Manual indexing and lack of automation.Limited event tracking and analytics.Basic autocomplete without custom recommendations.Need for personalization and similar product suggestions.Integration of facets into the front end. Optimization and scalability concerns.

Challenges Identified

So maybe something like using a cron job to schedule a daily update

as mentioned with the diagram and prior slides. we have a need to set up robust efficient indexing updates for both new items and updated items. there are several approaches we could take from running daily csv imports and than searching through date ranges. maybe using a data proccessing software like apache. or maybe just with a custom script. Its possible we could build off the third party integrations running php scripts that upload new products and have them push to our index also though.

Top Priority – Automated and Accurate Indexing

In addition to great analyics for our own use they would also be great for upgraded reccomendations and related products and re-ranking/other personlization.

We would like the most robust analytics we can get so we'd be looking to add event listeners to every action possible for the most part.

Robust Event Tracking and Analytics

Fleshing out the backend of our index configurations by properly implementing logic like re-ranking based on stock status, proper searhcable attributes, typo tolerance, synonyms, facets etc..Testing and refining search configurations.

Enhanced Autocomplete with Custom Recommendations

As a part of the upgraded css front end leftmost column allowing potentially for a faceted search and other custom recommendations from usImproving product filtering and navigation.Enhancing the front end for a cleaner and more intuitive interface.

Integration of Facets into the Front End

We need to ensure once we're live were not breaking the website, causing downtime, not properly displaying results etc.. creating a robust system of 250,000 products, potentially daily updates, and a high user volume.Code optimization for indexing and front-end performance.Batch operations and efficient data handling.Monitoring and profiling for bottleneck identification.Designing solutions that can scale with increased data and user load.Future-proofing the system for long-term growth.

Optimization and Scalability

Have to ensure were not introducing vulnerabilities into the system, or sharing our or user data.

Security and Data Protection

Defining what the Algolia team will handle vs. internal tasks.

Roles and Responsibilities

Timeline for each prioritized feature.Key milestones and deadlines.

Proposed Timeline and Milestones

Preferred communication channels (e.g., email, Slack).Regular meeting schedules.Points of contact from both teams.

Communication and Collaboration Plan

Invite questions and feedback from the Algolia team.Address any concerns or clarifications.

Questions and Next Steps

Finalize the plan based on feedback.Set up the initial kickoff meeting.Assign tasks and set up collaboration tools.

Immediate Next Steps

1. api code2. api get request return 3.current index4. template integrations5. backend coreware info, facets, 6.7.8.9.10.

Document list

1. hosting the scripts? servers etc..2.testing other api calls 3.4. 5. 6.7.8.9.10.

questions to answer

InvokeHTTP (Get All Product IDs)SplitJSON (Split Product IDs)InvokeHTTP (Get Full Product Info)EvaluateJsonPath (Extract Fields)RouteOnAttribute (Filter by Date Range)Extract UPCsPrepare UPC ListExecuteProcess (Run Custom Script)

current update solve