Business Need
Terapeak is a leading provider of data analytics for e-commerce marketplaces such as eBay, Amazon, and Alibaba. They have a suite of Software as a Service (SaaS) solutions and tools for online sellers to make sense of the complex dynamics of supply & demand, pricing trends, and market opportunities. Customers of Terapeak gain insight into their marketplace and can better take advantage of the opportunities to grow their businesses. Leveraging the business intelligence derived from the data, merchants can know where to get the most affordable merchandise, and what prices they should set their products to get the sales, and which e-commerce marketplace they should be selling on, while maximizing their profits.
As a Big Data company, Terapeak faces technical challenges of making billions of product listings, categories, sales transactions, web analytics events and aggregations available in their application platforms reliably and efficiently. Terapeak analyzes more than 5 billion transactions per year and over $70 billion in online consumer spending to provide marketplace intelligence to customers ranging from entrepreneurs to Fortune 500 companies. The success of their customers depends on their ability to acquire, analyze and provision data in a timely and reliable manner, and the future of Terapeak relies on their ability to innovate faster and execute better than their competitors.
To help them achieve their business and technology goals, Terapeak engaged Zettalytics to be responsible for all aspects of their data asset. This includes data acquisition, management, design, architecture, quality, and storage as well as the operational processes required to support Terapeak-branded customer-facing applications in a SaaS based application.
Technology Solution
Terapeak need technology solutions to help them ingest and process enormous amounts of data each day, both from batch files FTP’ed from their data partners as well as from APIs in real time. They also require a robust application platform to support the sophisticated analytics and data visualization requirements. We provided technology leadership and strategy to Terapeak’s entire engineering organization, including product development, data engineering and operations. Working side by side with the internal Terapeak team, Zettalytics consultants helped to architect, design and implement a number of key solutions to address performance, scalability, reliability, availability, agility and interoperability.
At the center of these solutions is a Unified Data Platform leveraging Hadoop, Spark, Spark Streaming, and Kafka for both batched and real-time ETL of tremendous volume of data. The new highly scalable architecture significantly improved customer experience by directly addressing data quality, timeliness and application reliability issues. Operational best practices introduced by Zettalytics consultants resulted in significant reduction in data and systems outages, and helped to improve the overall resiliency of a complex system. Our role in recommending and migrating the search platform from proprietary RDBMS and Solr to ElasticSearch contributed to a dramatic improvement in keyword search and real-time aggregation performance.
Zettalytics also worked closely with Terapeak’s product management and development teams to design the most powerful front-end analytics and data visualization application for e-commerce marketplace data using the latest technology stack, including ReactJS and RESTful Java APIs. We helped Terapeak to define and establish a new Agile development process that resulted in more predictable and reliable software releases. Together with the inhouse devops team, Zettalytics supported the infrastructure migration from a proprietary physical data center to the Cloud hosted at Amazon Web Services, thereby taking advantage of a scalable, reliable, and secure global computing infrastructure to host Terapeak’s world class application that is currently used by more than 500,000 users internationally.
Technology Stack:
- S3 and Hadoop for data staging
- Spark for data ETL
- Kafka for data queue
- Postgres for Data Warehouse
- ElasticSearch for indexing and search
- ReactJS for front-end application
Benefits
With the expertise and professionalism of Zettalytics consultants, Terapeak successfully implemented a new, highly scalable and reliable unified big data platform for processing hundreds of millions of rows of data per day. The resulting new product we helped to build allows e-commerce merchants to get better understanding of what products are currently top sellers, who their competitors are, and what prices are currently being offered for the products. This intelligence enable them to come up with better selling strategies, and identify any opportunities to further maximize their profits.
By leveraging Zettalytics’ knowledge and expertise in big data strategy and technology, Terapeak is able to solidify their position as a global leader in e-commerce analytics.
