Bulk Updates of Product Entities for a Large Catalog
A client faced significant challenges with their ERP integration for managing product data on their eCommerce platform. The ERP system was an outdated legacy version, lacking APIs for seamless data sharing. Instead, product additions, updates, promotions, and stock updates were provided via XML flat files. Additional files were used for localized attributes and brand mappings.
Problem
A client faced significant challenges with their ERP integration for managing product data on their eCommerce platform. The ERP system was an outdated legacy version, lacking APIs for seamless data sharing. Instead, product additions, updates, promotions, and stock updates were provided via XML flat files. Additional files were used for localized attributes and brand mappings.
However, the ERP-generated flat files presented two critical challenges:
Incompatibility with Magento’s Default Import Feature: The flat files were not formatted for direct processing in Magento.
Full Data Imports: The ERP system was incapable of sharing only delta changes, resulting in the complete catalog (over 300,000 items) being shared daily. This required significant processing time for updates related to inventory, promotions, and other attributes.
The client’s primary requirement was to ensure all data processing, including merchandising updates, indexing, and file generation for third-party activities, was completed before 6 AM daily.
Client Requirement
The client requested a custom module to
Process all flat files daily in an optimized manner
Ensure updated products, promotions, and inventory were visible to customers promptly at the start of the day.
Provide robust monitoring, logging, and error-handling capabilities.
Solutions
We developed a bulk processing module tailored to handle the client’s requirements efficiently. Leveraging direct database updates and a custom queue mechanism, the module ensured timely and accurate data processing.
Key features implemented
A centralized admin interface was implemented to monitor flat file processing in real time, backed by detailed logging, email notifications, and robust error-handling to ensure data accuracy and operational reliability. The solution also optimized large-scale updates through direct database operations, custom queues, MSI-aware inventory handling, automatic indexer updates, and intelligent promotion mapping from ERP to Magento.
File Monitoring and Administration
A built-in admin interface to monitor the progress of processing for each of the 10 flat files uploaded to the FTP server.
Real-time tracking of processing status to ensure timely completion.
Comprehensive Logging and Notifications
Detailed logs for every update made to the product catalog, capturing both the updated product attributes and values.
Email notifications for missing mandatory files, issues in uploaded files, and successful processing.
Error Handling and Flexibility
Custom error-handling mechanisms for resolving data issues in flat files.
Standalone Magento commands to allow manual execution of file processing as needed, providing flexibility in case of unexpected scenario
Optimized Processing
A custom queue mechanism to prioritize critical updates and ensure sequential processing.
Use of direct database updates to minimize processing time for large datasets.
Indexing and Multi-Source Inventory (MSI) Considerations
Automatic programmatic updates of dependent indexers as required.
Inventory updates accounting for MSI to ensure accurate stock allocation across multiple sources.
Promotion Handling
Custom handling for grouped promotions shared via ERP, segregating them into product-specific promotions in Magento.
