Challenge

• Customer has the Hubspot and MySQL systems working as their system of records for customer data. Hubspot is responsible for storing the data for the marketing campaigns of programs/students and MySQL acts as the data warehouse.

• With increases in students, campaigns, organisations and contacts added to the system everyday from various interaction points of the organisation, data sync from one system to another had become cumbersome.

• The existing framework and design failed to successfully carry out these operations, leaving gaps and errors in data sync due to data loads and architectural challenges.

Strategy and Solution

• Customer collaborated with NJC Labs to create a new, integrated system with reusable and efficient architectural design.

• The existing framework on C# was replaced by modern APIs and converted to reusable assets.

• To deal with the volume of data that’s sent in for the data sync, multiple solutions with the core value of reusing the data was developed. Multiple paginations and efficient logics were written for complex transformations in between the sync.

• An efficient error-notification system was developed to notify the stakeholder of any issues. Notifications were broken down and formulated with the intention of understanding the issues in layman’s terms.

• Additionally, alerts were set to track the Memory/CPU usage for continuous monitoring avoiding dependencies.

Transformation

• The task was completed successfully implementing a design first, reusable modern API.

• Over 16 objects/parameters of over 1.5 million - 2 million data were successfully synced in a bi- directional approach from hubspot to data-warehouse and vice-versa.

• Objects carrying more data counts were efficiently built using different layers of paginations avoiding data bulk issues and were effortlessly synced.

• Error notifications on every errors and efficient logging at all traces helped the team to effortlessly track and look into the concerns if any.

Result

• Reusable API formation were created which could be further used for future integrations to the respective systems.

• Continuous data sync between Hubspot and DWH almost eliminated errors.

• Debug/Monitoring time has been reduced considerably with the design of an efficient logging/notification system.