Anchor: Navigation

Retail Issue 4.0 - February 2025 Follow the Stream Your Dedicated Community Content Scroll 

Executive's Brief
Customer Experience
Architect's Blueprint
Developer's Desk
Innovation & Research

The Year of DSP: Data Streaming in 2025 & Shift Left

Welcome to the fourth issue of Follow the Stream! To kick off 2025, this edition is dedicated to the concept of "Shift Left”—a data processing and governance paradigm that moves data processing closer to the source and ingests fresh trustworthy data into any downstream consumers. Covering data streaming predictions and trends for the upcoming year, we explore how Shift Left architecture is revolutionizing data quality and providing reusable data products for both operational and analytical use cases in financial services.
See Industry Updates

Data Streaming Fundamentals

Learn More

Data Streaming Crash Course

Let's explore the basics of Shift Left & Data Streaming Platforms.

Heading 1

Subtitle 1

Heading 2

Subtitle 2

Heading 3

Subtitle 3
Shift Left is a rethink of how to circulate, share, and manage data. Along with a data streaming platform, it enables organizations to build data once, build it right, and reuse it anywhere within moments of its creation.
Data governance plays an important role ensuring data quality, security, and compliance across the data lifecycle, while data products, built around real-time streams, facilitate reusable, high-quality data assets for efficiency and collaboration.
Subtitle 1
Subtitle 2
Subtitle 3
Anchor: General-Updates

Industry in Motion: General Updates

Download Ebook for Free
Available in German, French, and Spanish!

3 Data Engineering Trends Riding Apache Kafka®, Apache Flink®, and Apache Iceberg™

The Apache Kafka, Flink, and Iceberg communities are continuously evolving, offering innovative ways for engineers to manage and process data at scale. From re-envisioning microservices with Flink streaming applications to enabling real-time AI model applications, these tools are reshaping data integration. With strong community contributions, especially to Iceberg, data governance and real-time analytics are set to accelerate, revolutionizing how businesses manage their data infrastructure.
Read Article

The Data Streaming Landscape 2025

“Data streaming is a new software category. It has grown from niche adoption to becoming a fundamental part of modern data architecture, leveraging open source technologies like Apache Kafka® and Apache Flink®. With real-time data processing transforming industries, the ecosystem of tools, platforms, and cloud services has evolved significantly.” Kai Waehner, Confluent’s Field CTO, “explores the data streaming landscape of 2025, analyzing key players, trends, and market dynamics shaping this space.”
Read Blog

Ushering in a New Era of Data Streaming Confluent Recognized as a Challenger in 2024 Gartner® Magic Quadrant™ for Data Integration Tools

Recognized as a Challenger in Gartner’s 2024 Magic Quadrant for Data Integration Tools, Confluent predicts that the future of data integration will focus on universal data products and event-driven flows, with real-time streaming bridging gaps between operational and analytical data. This shift will drive innovation in areas like fraud detection, personalized experiences, and supply chain optimization, making data streaming the standard for faster, more efficient decision-making.
Read Blog

New in Confluent Cloud: Extending Private Networking Across the Data Streaming Platform

Confluent’s Q4 2024 launch wrapped up the year with a host of powerful features paving the way for even more innovation in 2025. Focusing on delivering private networking and enhanced security across the data streaming platform, key features include mTLS authentication, AWS PrivateLink for Schema Registry, Apache Flink® UDFs for custom stream processing, and WarpStream Orbit for seamless Apache Kafka® migrations.
Read Blog
Back to TopSign Up for Updates
Anchor: Executives-Brief

Executive's Brief: Data Strategy & Leadership

Predictive Analytics: How GenAI and Data Streaming Work Together to Forecast the Future

Predictive analytics, enhanced by generative AI and data streaming, is transforming decision-making in retail. By utilizing historical data and machine learning models, predictive analytics allows retailers to forecast customer behavior, demand trends, and sales outcomes with greater precision. Generative AI takes this a step further by generating diverse scenarios, filling data gaps, and continuously adapting to real-time data for more accurate, dynamic forecasting.
Read Blog

Shift Left: Unifying Operations and Analytics with Data Products

The need for high-quality business data is greater than ever, so preventing and mitigating bad data—across the entire business—has become a critical capability. Shifting data processing and governance “left” eliminates duplicate pipelines, reduces the risk and impact of bad data at the source, and leverages high-quality data products for both operational and analytical use cases.
Read Ebook

Good Teams Manage Kafka – Efficient Teams Use Confluent

Managing open source Apache Kafka® in-house or through basic cloud-hosted services can lead to significant costs, operational challenges, and downtime risks, especially under tight engineering budgets. Using a fully managed, cloud-native platform can help reduce these challenges, enabling teams to focus on strategic initiatives while improving cost efficiency and accelerating time to market for real-time data projects.
Download White Paper
Available in German, French, Spanish, and Portuguese!
Watch on YouTube

Hungry for Data – Huel Decouples Infrastructure with Data Streaming 

Huel, a rapidly expanding food subscription service, transformed its infrastructure by adopting an event-driven microservices powered by Apache Kafka®. This shift allowed Huel to decouple its systems and leverage real-time data, enhancing its subscription platform and inventory management. As a result, Huel optimized operational efficiency, enabling smarter decision-making and personalized customer experiences. 
Read Article
Back to TopSign Up for Updates
Anchor: Customer-Experience

Data Streaming for Customer Experience

How Data Streaming Elevates the Omnichannel Customer Experience

To realize its vision of a frictionless omnichannel experience, Thrivent invested in an agile data architecture that supports real-time data ingestion and processing. Alongside digital services consulting firm Improving, Thrivent discussed the challenges of building a next-gen data streaming architecture and best practices for developing reusable APIs to ensure a seamless customer experience across channels.
Watch Webinar

Data Streaming’s Role in Shaping and Scaling the Customer Experience for Fnac Darty

Fnac Darty uses data streaming to deliver real-time updates on pricing, inventory, and customer navigation events, ensuring a seamless and responsive shopping experience. By integrating siloed data and enabling instant access to actionable insights, the company can better understand customer intentions and optimize operations to meet their needs. This approach enhances agility, improving customer satisfaction and loyalty.
Read Article

Vimeo Revolutionizes Real-Time Experiences for Over 260 Million Users with Confluent

Vimeo uses data streaming to deliver real-time insights that improve user experiences, optimize video playback, and enable agile decision-making. By utilizing adaptive bitrate streaming, Vimeo dynamically adjusts video quality based on real-time network conditions, ensuring smooth playback without buffering. This real-time approach empowers the video platform to iterate faster, monitor campaigns, and make data-driven product decisions.
Read Case Study
Back to TopSign Up for Updates
Anchor: Architects-Blueprint

Architect's Blueprint: Data Systems Design

The Streaming Data Quality Guidebook

Maintaining high data quality is critical, especially with the rise of AI/ML, where poor data can lead to system failures, inaccurate decisions, and costly outages. By implementing best practices for stream governance—identifying schema issues, establishing data contracts, and adopting decentralized data ownership—organizations ensure the integrity of their data streams and significantly reduce risks.
Download Guidebook
Available in German, French, Spanish, and Japanese!

From Batch and Lakehouse to Real-Time Data Products with Data Streaming

“The Shift Left Architecture enables a data mesh with real-time data products to unify transactional and analytical workloads with Apache Kafka, Flink, and Iceberg. Consistent information is handled with streaming processing or ingested into Snowflake, Databricks, Google BigQuery, or any other analytics/AI platform to increase flexibility, reduce cost, and enable a data-driven company culture with faster time-to-market building innovative software applications.”
Read Blog

Real-Time Data Streaming for Smart Warehouses

Data streaming empowers retailers to revolutionize warehouse operations by enabling real-time inventory tracking, predictive maintenance, and optimized resource allocation. Real-time data enhances automation, reduces errors, and ensures timely fulfillment to meet customer demands, such as same-day delivery. By breaking down data silos and enabling actionable insights, data streaming supports agile and customer-centric logistics in the retail sector.
Read Blog
Anchor: must-read

Data Streaming Journey at Migros | Data Products, Observability, Event-Driven Microservices, IoT

Migros, one of the largest retailers and employers in Switzerland, started its data streaming journey in 2016 for tracking customer events across touchpoints. In an effort to continue optimizing its global supply chain, the supermarket chain leveraged data streaming for “customer master data management, product and store information updates, and synchronization with Migros Online.” By 2022, Migros kicked off the Enterprise Integration Platform, enabling data products across the organization.
Read the full story and many more in the new book by Kai Waehner: The Ultimate Data Streaming Guide. Download is available for free.
Download Ebook for Free
Available in German, French, and Spanish!

How the Retailer Intersport uses Apache Kafka® as Database with Compacted Topic

Intersport uses Apache Kafka's compacted topics to efficiently manage and query real-time data across global operations. By leveraging this feature, the retailer ensures accurate, up-to-date stock information throughout its supply chain, enabling seamless integration between systems like ERP, POS, and inventory management. This approach helps streamline operations, improve data consistency, and scale flexibly during high-demand periods, such as holidays.
Read Blog
Back to TopSign Up for Updates
Anchor: Developers-Desk

Developer's Desk: Building Applications

How Apache Iceberg and Flink Can Ease Developer Pain

Although developing stateful applications can be chaotic due to complex upstream and downstream interactions, tools like Apache Iceberg and Apache Flink help simplify this process. Iceberg optimizes data querying by defining efficient table structures, while Flink enables real-time data processing, improving speed and reliability. Together, they provide a robust framework that enhances both the efficiency and reliability of stateful application development.
Watch Podcast

Shift Processing and Governance Left: A DSP Product Demo

“Shift left" is gaining traction as a trend by addressing the increasing need for high-quality business data. This approach solves common challenges like redundant datasets, poor data quality, and high costs associated with data warehouses and data lakes by cleaning and aggregating data closer to the point of data generation.
Watch Demo

Handling the Producer Request: Kafka Producer and Consumer Internals | Part 2

The Apache Kafka® producer request process involves a series of stages, including handling by network threads, queuing, data validation by I/O threads, and eventually writing to disk. Throughout this journey, configurations such as buffer sizes, thread counts, and flush intervals play key roles in optimizing performance. Monitoring metrics help track the efficiency of each step, ensuring that the data is properly processed and replicated for durability.
Read Blog

Key Concepts of a Schema Registry

Schema Registry is a tool that manages and enforces data schemas in the Apache Kafka ecosystem, ensuring data compatibility and quality across applications. Understanding key concepts like schemas as contracts, the role of Schema Registry, schema workflow, and integration with client applications can help developers enhance system reliability and overall data streaming capabilities.
Watch Free Course

How to Add a Connector to Confluent Cloud

Kafka Connectors provide a way to get data flowing between sources and sinks, and Confluent Cloud. Choosing between available options like fully managed connectors for easy setup, or custom connectors that suit specific requirements, ultimately depends on the required level of control and specific data pipeline goals.
Watch on YouTube

Learn About Data Streaming With Apache Kafka® and Apache Flink®

High-throughput low latency distributed event streaming platform. Available locally or fully-managed via Apache Kafka on Confluent Cloud.

High-performance stream processing at any scale. Available via Confluent Cloud for Apache Flink.

Explore Developer Hub

Request Flink Workshop or Tech Talk

Back to TopSign Up for Updates
Anchor: Innovation-Research

Data Streaming for Innovation & Research

A New Era in Dynamic Pricing: Real-Time Data Streaming with Kafka and Flink

Dynamic pricing is “a flexible model that adjusts prices based on real-time market changes to enable real-time responsiveness to demand, competitor prices, and customer behaviors.” Leveraging data streaming allows companies like AO to “to seize pricing opportunities and align closely with customer expectations,” increasing customer conversion rates by 30%.
Read Blog

Real-Time Data Streaming is Key to a Successful Christmas Campaign

Data streaming is the “invisible but essential technology” for retailers, especially during critical times like the Christmas season. By analyzing customer behavior in real time, businesses can launch timely flash sales, suggest complementary products, and blend omnichannel discounts, creating a seamless shopping experience that drives conversions and boosts customer loyalty throughout the holiday rush.
Read Article
Back to Top

Join the community

Sign up for updates below and check out previous issues!
Issue 1.0
Issue 2.0
Issue 3.0
Sign Up for Updates
X
LinkedIn
GitHub

Share content suggestions and new uses cases in the Comments Section

How Data Streaming Elevates the Omnichannel Customer Experience

To realize its vision of a frictionless omnichannel experience, Thrivent invested in an agile data architecture that supports real-time data ingestion and processing. Alongside digital services con...

Schema Registry 101

In the event-driven world, schemas are essential as they can enforce the shape of the data and act as contracts between producers and consumers. The Schema Registry can manage those contracts and a...

Handling the Producer Request: Kafka Producer and Consumer Internals | Part 2

In this post, the second in the Kafka Producer and Consumer Internals Series, we follow our brave hero—a well-formed produce request—which is on its way to be processed by the broker and have its d...

Shift Left: Unifying Operations and Analytics With Data Products

ELT pipelines can lead to duplicates and high costs. A 'shift left' strategy improves this. We discuss making data available in operational, analytical, and hybrid systems with streams, tables, sch...

New Paradigm to Shift Left

Shift Left is derived from the software engineering principles of Shift Left Testing where tests are performed earlier in the development lifecycle to improve software quality, accelerate time to m...

New Paradigm to Shift Left: Why is Shift Left Important?

Shift Left is derived from the software engineering principles of Shift Left Testing where tests are performed earlier in the development lifecycle to improve software quality, accelerate time to m...

Data Governance

What is data governance, and why does it matter? Learn how data governance works, steps to real-time governance, benefits, challenges, and considerations.

Data Governance

What is data governance, and why does it matter? Learn how data governance works, steps to real-time governance, benefits, challenges, and considerations.

The Streaming Data Quality Guidebook

Learn Stream Governance best practices to protect downstream systems and applications from poor data quality with this comprehensive guide.

3 Data Engineering Trends Riding Apache Kafka®, Apache Flink®, and Apache Iceberg

The Apache Kafka, Flink, and Iceberg communities are continuously evolving, offering innovative ways for engineers to manage and process data at scale. From re-envisioning microservices with Flink ...

Confluent Recognized as a Challenger in 2024 Gartner® Magic Quadrant™ for Data Integration Tools: Ushering in a New Era of Real-Time Data Streaming

Recognized as a Challenger in Gartner’s 2024 Magic Quadrant for Data Integration Tools, Confluent predicts that the future of data integration will focus on universal data products and event-driven...

New in Confluent Cloud: Extending Private Networking Across the Data Streaming Platform

Confluent’s Q4 2024 launch wrapped up the year with a host of powerful features paving the way for even more innovation in 2025. Focusing on delivering private networking and enhanced security acro...

Predictive Analytics: How Generative AI and Data Streaming Work Together to Forecast the Future

Predictive analytics, powered by data streaming and GenAI, is transforming risk management, fraud detection, and investment strategies. By analyzing historical and real-time data with machine learn...

Shift Processing and Governance Left: A DSP Product Demo

“Shift left" is gaining traction as a trend by addressing the increasing need for high-quality business data. This approach solves common challenges like redundant datasets, poor data quality, and ...

How Apache Iceberg and Flink Can Ease Developer Pain

Although developing stateful applications can be chaotic due to complex upstream and downstream interactions, tools like Apache Iceberg and Apache Flink® help simplify this process. Iceberg optimiz...

The Ultimate Data Streaming Guide | Concepts, Use Cases, and Industry Stories for Retail & E-commerce

The long-awaited book by Kai Waehner, Field CTO at Confluent, is finally here! Fresh off the press, it showcases the usage of data streaming across industries, including retail. He not only focuses...

Good Teams Manage Kafka – Efficient Teams Use Confluent

Managing open source Apache Kafka® in-house or through basic cloud-hosted services can lead to significant costs, operational challenges, and downtime risks, especially under tight engineering budg...

Hungry for Data – Huel Decouples Infrastructure with Data Streaming

Huel, a rapidly expanding food subscription service, transformed its infrastructure by adopting an event-driven microservices powered by Apache Kafka®. This shift allowed Huel to decouple its syste...

Data Streaming’s Role in Shaping and Scaling the Customer Experience for Fnac Darty

Fnac Darty is a leading European retailer specializing in entertainment and leisure products, consumer electronics, domestic appliances and services. Headquartered near Paris, the company employs n...

Vimeo Revolutionizes Real-Time Experiences for Over 260 Million Users with Confluent

How Vimeo moved from batch to stream processing using Confluent for real-time insights, personalized user experiences, driving massive growth and profitability.

Blog: Real-Time Data Streaming for Smart Warehouses

Data streaming empowers retailers to revolutionize warehouse operations by enabling real-time inventory tracking, predictive maintenance, and optimized resource allocation. Real-time data enhances ...

What is a Data Streaming Platform (DSP)?

Learn how a data streaming platform (DSP) enables organizations to capture, store, and process data as a continuous flow of real time events.

Real-time Data Streaming is Key to a Successful Christmas Campaign

El streaming de datos en tiempo real impulsa la personalización y las ventas durante la campaña de Navidad, según el informe Data Streaming Report 2024 de Confluent.

Real-Time Click Stream Analytics for Upselling & Cross-Selling

Real-time data streaming and clickstream analytics empower businesses to upsell and cross-sell by acting on customer interactions in real time and making personalized recommendations before custome...

The Shift Left Architecture - From Batch and Lakehouse to Real-Time Data Products with Data Streaming

Blog about architectures, best practices and use cases for data streaming, analytics, hybrid cloud infrastructure, internet of things, crypto, and more

The Data Streaming Landscape 2025

“Data streaming is a new software category. It has grown from niche adoption to becoming a fundamental part of modern data architecture, leveraging open source technologies like Apache Kafka® and A...

Data as a Product

In many applications, data streams are treated as a byproduct of the system. However, the most successful businesses recognize that when data is treated as a first-class product, it can drive their...

Data Mesh 101: Data as a Product

Data as a product, the second principle of data mesh, further delineates the data deliverables for which each domain team is responsible. Follow along as Tim Berglund (Vice President, Developer Rel...

A New Era in Dynamic Pricing: Real-Time Data Streaming with Apache Kafka and Flink

Blog about architectures, best practices and use cases for data streaming, analytics, hybrid cloud infrastructure, internet of things, crypto, and more

How to Add a Connector to Confluent Cloud

Kafka Connectors provide a way to get data flowing between sources and sinks, and Confluent Cloud. Choosing between available options like fully managed connectors for easy setup, or custom connect...

Build Customer Loyalty Programs with Data Streaming

In this tutorial, learn how to track customers' purchasing patterns, generating tailored rewards for a loyalty scheme using Confluent, with step-by-step instructions and examples.

Shift Left to Write Data Once, Read as Tables or Streams

Shift Left is a rethink of how we circulate, share and manage data in our organizations using DataStreams, Change Data Capture, FlinkSQL and Tableflow. It addresses the challenges with multi-hop a...

Shift Left to Write Data Once, Read as Tables or Streams

Shift Left is a rethink of how we circulate, share and manage data in our organizations using DataStreams, Change Data Capture, FlinkSQL and Tableflow. It addresses the challenges with multi-hop a...

What is a Data Streaming Platform?

A Data Streaming Platform (DSP) is composed of six major inter-related components, covered in detail in this video. Each component has evolved over time to servce specific needs in the data streami...

How the Retailer Intersport uses Apache Kafka® as Database with Compacted Topic

Intersport uses Apache Kafka's compacted topics to efficiently manage and query real-time data across global operations. By leveraging this feature, the retailer ensures accurate, up-to-date stock ...

What is a Data Streaming Platform?

A Data Streaming Platform (DSP) is composed of six major inter-related components, covered in detail in this video. Each component has evolved over time to servce specific needs in the data streami...