Client Requirements
Client Overview
Client is the leading cross-channel retailer in the "Green Industry," with over 130 brick-and-mortar stores and an extensive online presence.
Spryker Cloud Commerce OS was selected as e-commerce platform to support online business which is currently operating four online stores with approximately 45.000 SKUs per store.
Given the scale of Client’s operations and ambitions for further growth, we needed to develop a solution that addresses their specific integration requirements, ensuring seamless scalability and uninterrupted business operations.
Challenges in Data Processing
Efficient data processing is crucial for maintaining the integrity and performance of an e-commerce platform. This section outlines the significant challenges faced when processing large datasets within Spryker Cloud Commerce OS.
Spryker Cloud Commerce OS Performance Degradation
Handling large datasets (50k+ records) using the Spryker Standard Implementation leads to substantial performance degradation. This issue was notably observed during the project when we first run the integration performance tests using Spryker Standars Implemention.
Key challenges and observations:
- Extended Import Times: Import times increase significantly as the number of records grows, primarily due to row-by-row processing
- Processing Bottlenecks: The high volume of events created related to the import, sometimes duplicated, acts as a bottleneck, slowing down the entire system. This issue not only impacts import processes but also prevents timely processing of other events
- Rabbit MQ & Jenkins Stability: Resource-intensive operations have led to occasional crashes of Rabbit MQ and Jenkins.
- Publishing Delays: There are delays in updating product prices and stock availability on the storefront, resulting in discrepancies where customers may see outdated information.
Business Impact
The repercussions of observed challenges and inefficient data processing extend beyond technical performance and resource wastage, affecting business operations and customer satisfaction and has following impact:
Overview of Improved Mechanism
We have developed a robust mechanism that significantly enhances performance and scalability. This section outlines the details of our improved import and processing
Improved Mechanism
To address Client’s Integration Requirements and overcome the limitations of the Spryker Standard implementation, we have decided to cover the end-to-end process using two key components: the Data Importer and Data Publisher. These components consist of smaller steps to enhance error handling and processing efficiency.
This approach allows us to refine and enhance performance while effectively implementing project-specific adjustments in the future, catering not only to this specific Client but also to any customer with similar requirements.
Data Importer: Price and Stock Overview
Receive data from SAP via Glue API: data is received and written into import table
Batch Size Parameter: parameter for defining chunk size, in our case the chunk size is defined to 50k records
Jenkins Job execution: The Jenkins job executes a command to start the process. All subsequent steps are executed in a loop until all records in the import table are processed. In case of an error, the loop exits, and the error message is displayed in the Jenkins output console.
Mark the records for processing: Assign UUIDs to records in the import table, based on the chunk size. All subsequent steps and logic are applied only to that specific record set, i.e., chunk.
Execute Calculations: This step prepares the data and updates the import table with information such as availability, offer references, foreign keys, etc. This allows us to leverage only import table in following steps reducing the probability of database deadlocks on Spryker tables.
Import Data into Spryker Tables: The enriched data is imported into Spryker tables. All the necessary data is taken from the import table, ensuring no additional calculations are required during this step.
A Comparative Analysis
Our innovative implementation has demonstrated a substantial enhancement in processing times. This acceleration is not only important for maintaining timely product updates but also ensures uniformity across the storefront.
Comparative Analysis
In test case, we evaluated the system’s performance with a substantial data set of 758k records. The results were impressive.
If we extrapolate from our test case, to achieve Client’s goal of importing 6 million records, using the Spryker Standard Implementation it would approximately take a full day (24 hours). However, with our Improved Implementation, the same task could be accomplished in a mere 80 minutes.
This contrast underscores the efficiency and effectiveness of our Improved Implementation, making it a powerful tool for large-scale data processing. Notably, we are further looking into optimizing the overall process thus potentially making the overall time even shorter.
Conclusion
Our implementation was tested and verified on Client project, and resulted with a significant improvement in processing times, which is crucial for maintaining product updates and ensuring consistency across the storefront.
Summary of Findings
Our changes to the data processing in Spryker Cloud Commerce OS have led to a big 94% increase in overall performance. That’s 18 times faster than Spryker Standard Implementation. This means businesses can handle more data, update their online stores quicker, and give their customers the most up-to-date information.
By getting rid of the less efficient parts of the Spryker Standard Implementation we’ve cut down on wasted resources, aligning with Spryker’s commitment to sustainability. By optimizing resource usage, we’re contributing to a more sustainable and energy-efficient e-commerce ecosystem.
Additionally, our solution is flexible. It’s designed to support business growth and change, making sure it stays efficient and reliable in the long run. Whether we are supporting a small business planning to grow, or a large business looking to optimize, our solution can adapt to their needs.
Furthermore, this mechanism will be further improved and adjusted according to specific project needs, ensuring a tailored fit for every unique client and requirement.
In short, our improved mechanism for Spryker Cloud Commerce OS is about more than just better data processing. It’s about helping businesses work more efficiently, serve their customers better, and grow.
Unlock the Potential of Your Digital Commerce
Discover how Xiphias can help you build high-performance, scalable commerce platforms. From Spryker to Pimcore, our solutions empower your business to deliver seamless experiences across web, mobile, and enterprise systems. Give your team the tools to innovate faster and achieve exactly what your customers need.
Get in Touch