In the first issue of Follow The Stream, we invite you to explore the fundamentals of data streaming, what it is, and how it empowers businesses. After looking through the Crash Course, feel free to scroll through the whole issue or jump to role-specific sections using the navigation menu above. Enjoy!
Now that we have the basics down, let’s dig a little deeper.
Heading 1
Subtitle 1
Heading 2
Subtitle 2
Heading 3
Subtitle 3
Data in Motion is the foundation for everything at Confluent. Check out the links below to get an understanding of how real-time data processing powers business productivity and innovation:
Subtitle 1
Subtitle 2
Subtitle 3
Anchor: General-Updates
Industry in Motion: General Updates
Forrester Evaluates Data Streaming Platforms
A data streaming platform is a software platform that empowers businesses to stream, process, connect, and govern real-time data streams. It turns data mess of batch-oriented, point-to-point connections into data value. Forrester's evaluative report on streaming data platforms reflects growing investments, interest, and innovation in the category. It recommends that data-streaming customers look for providers that offer a breadth of core capabilities, development tools that support fast change, as well as efficient and fault-tolerant scalability. In their evaluative report, Confluent received the highest score possible in 10 of the 21 criteria that vendors were evaluated on, including:
Current offering: connections, movement, management, and fault-tolerance
Strategy: vision, innovation, roadmap, partner ecosystem, community
The Data in Motion Tour 2024 is stopping by your city! Join us for an opportunity to network with your peers and ecosystem partners, hear directly from customers, learn from our team of Apache Kafka® experts, and roll up your sleeves in interactive demonstrations and hands-on labs.
Upcoming dates: September 26th: Paris - RSVPOctober 9th: Vienna - RSVPOctober 15th: Madrid - RSVP
5 Data Engineering Predictions to Unlock Greater Data Value
Five predictions for the data engineering field in 2024 include:
Generative AI Integration Across Applications
Early Data Governance Implementation
Wider Adoption of Apache Flink® for Stream Processing
Enhanced Cloud-Native Apache Flink® 2.0
Mainstream Adoption of Data as a Product
These forecasts highlight the main focus for many data engineers this year. Considering rapid shifts in the technological landscape, data engineers will act as main architects of change with their expertise and creativity shaping the data infrastructures of tomorrow.
No Data = No AI. Data Streaming for Actionable Intelligence with AI
Real-time data is at the heart of successful AI integration. Besides cleaning and preparing data required to train the initial model, businesses have to consider data accessibility on the enterprise-wide level. With each successive AI effort more and more data is required, which leads to increased costs and complexity. However, when data is distributed across the organisation in real time as it is created, AI tools deliver maximum value and drive actionable intelligence. Data streaming gives AI applications:
After last year’s debut, Confluent is back with the virtual festival experience of the summer! Bigger, brighter, bolder — this event is a month-long celebration of the data streaming community and innovation happening around us. Get ready for outstanding headliners, merch raffle, and three stream-packed stages: • Re-Inventing Apache Kafka®• Flink on Stage - Feat. Query & The Squirrels• Building the Future Today - Data Mesh & Gen AI Register before July 14th for a chance to score an exclusive festival merch package!
Data Streaming Awards 2024: Nominations Are Now Open
The Data Streaming Awards is back for its third year! Designed to bring the data streaming community together, this one-of-a-kind industry award event recognizes organizations that are harnessing the power of this revolutionary technology to drive business and customer experience transformation. Do you know a company (even your own team) that is using data streaming technology to transform their business? Nominate them here.
Responsibility of Executives in Data Steaming Adoption
Over the last decade, more than 26,000 companies have decided that the solution they’ve been looking for is data streaming. However, even early adopters continue to face non-technical hurdles associated with organisational silos: Here is what technical leaders can do to realize the full potential of data streaming:
Align team or department objectives with broader company strategy
Decide when to use data streaming vs batch processing
Understand the data streaming ecosystem and required capabilities
Equip technical teams to evaluate data solutions based on business needs
Develop a strategic roadmap for prioritising data streaming use cases
Guide developer enablement and holistic implementation
Bonus tip: A key success factor is strategy centred around decentralised data ownership, selfservice, and standardisation.
How Data Streaming Improves Overall Equipment Effectiveness
Learn about data streaming applications within the manufacturing industry directly from Confluent’s Global Field CTO and data streaming expert, Kai Waehner. In this episode he explores use cases around predictive maintenance, explaining how to leverage sensor data that continuously stream data and use it to improve the overall equipment effectiveness (OEE). Kai also demonstrates how to embed AI and Machine Learning into streaming applications to ensure you can run predictive maintenance while the data is hot.
Although AI and machine learning (ML) have become mainstream in today's business world, truly innovative models depend on real-time data streaming for accuracy. “Data streaming is the central nervous system for data, while AI/ML algorithms are the brain.” –Tweet ThisTrusted data is especially crucial for user-facing applications such as automated self-service systems (chatbots), as they can revolutionise CX - answering inquiries faster than ever - or “produce coherent nonsense” resulting in repetitional risk.
How Data Streaming Empowers Connected Manufacturing and GenAI Solutions at Bosch
Bosch embarked on its Industry 4.0 journey over a decade ago, pioneering the digitization and interconnection of its global and customer plants. Today, these initiatives are combined with AI to deliver cutting-edge solutions for production scheduling, monitoring, and control. Connected manufacturing networks collect and feed large data sets to AI for evaluation, which empowers solutions like automated optical inspection. This data-driven approach is driving significant productivity gains, with annual increases surpassing six figures per plant. Recognized by the World Economic Forum as an Industry 4.0 Lighthouse, Bosch not only leads in AI development but also showcases tangible improvements at an already advanced Bursa plant:
In this blog post, Confluent’s Field CTO and data streaming expert Kai Waehner looked at trends in the manufacturing industry to analyse their impact and explore the industry’s relationship with data streaming. He suggests that current trends shaping the industry, including software-defined manufacturing and robotic automation, are only possible if enterprises “can provide and correlate information at the right time in the proper context.” Kai also discuses customer stories from worldwide manufacturers across industries:
BMW: From smart factories to the cloud
Michelin: From shop floor to customer service
50Hertz: From monolithic SCADA to cloud-native IoT
Siemens: From SAP ERP on-premise to Salesforce CRM in the cloud
Mercedes: From manual business processes to seamless customer 360
Transforming the Global Supply Chain with Data Streaming and IoT
According to IoT Analytics, eight key technologies are the future of the global supply chain. Each relies on different capabilities of real-time data streaming to enhance efficiency, reduce costs, or improve decision-making. For example, AI-enabled inventory optimization deals with the challenge of organizing millions of data points: “ Information from warehouses, department stores, suppliers, shipping, and similar inventory-related data sources must be correlated to maximize customer satisfaction and revenue growth and increase customer conversions.” AI optimization “automates, streamlines, and controls the in- and outbound inventory” to manage inventory levels more efficiently.
Harnessing GenAI in Manufacturing and Supply Chains
GenAI is revolutionising operations by enhancing efficiency and productivity “across the plan-make-deliver value stream." In planning, it consolidates insights for better demand forecasting and supply chain disruption mitigation while optimising inventory management. In manufacturing, GenAI boosts productivity by predicting failures, reducing defects, and supporting operators with AI-driven guidelines. For delivery, it automates documentation, ensures timely product transit, and communicates with customers, further accelerating warehouse design and production scenarios through digital twins.
Data Streaming for Enhanced Supply Chain Resilience and National Security Harmony
Resilient supply chains are crucial for the overall stability and security of nations. A key strategy for ensuring reliability throughout the entire supply chain is real-time data streaming. For example, “up-to-the-minute information on inventory levels, production status, and transportation conditions” enhances situational awareness, reducing risk and response time in times of crisis. Data in motion also helps prevent uncertainties and enables real-time decision making, both of which are critical for effectively mitigating disruptions.
Delivering Real-Time Predictive Maintenance with Data Streaming
Data streaming provides real-time visibility into the engine manufacturing process, allowing for immediate detection and correction of quality issues. By continuously capturing and analyzing data from manufacturing steps, predictive maintenance can be implemented to prevent costly downtime and extend equipment lifespan. This proactive approach ensures higher product quality, minimizes scrap, and optimizes maintenance schedules, ultimately increasing operational efficiency and reducing costs.
Invaluable data is hidden within every level of the supply chain, "from tracking inventory levels and monitoring production processes to analysing customer behaviour and monitoring logistics operations." Traditional data architectures in supply chains relied on batch-oriented, manual processes, which lead to inefficiencies, lack of real-time insight and processing bottlenecks. Modern data architectures prioritise real-time data handling, scalability, integration, and advanced analytics to overcome these challenges. Emerging trends transforming supply chain data management include cloud-based solutions, big data analytics, IoT, AI, blockchain, edge computing, and data visualisation tools. These technologies enable improved visibility, real-time insights, predictive analytics, cost savings, better decision-making, enhanced collaboration, and agility.
Real-Time Order Management: The Key to Streamlining Your Business Implementing a successful order management system (OMS) requires multiple integrations, which rely on consistent, up-to-date view of data at all times, as well as careful tailoring to specific businesses processes. To achieve a seamless end-to-end buying journey for customers, companies need to ensure data accuracy, order authenticity and fraud detection within their data pipelines. Stream processing enables you to build in-stream detection and analysis of order authenticity through pattern matching as the first layer of defense.
According to McKinsey, aftermarket sales are “vital to manufacturing strategies”. However, companies across industries are struggling to digitise and optimise processes in secondary markets. Lack of internal capabilities, data compatibility, and high quality data are just some of the challenges preventing manufacturers from “automated context-specific decision-making in real-time when it is relevant (predictive maintenance) or valuable (cross-upselling)”.To address challenges in aftermarket sales, Michelin transitioned from a complex Oracle BPM system to a microservices architecture enhancing real-time data integration and processing. By adopting cloud-native technologies, Michelin boosted operational efficiency and customer experience through reliable real-time data access.
The Data Streaming Revolution: Rise of the Kafka Heroes
A captivating comic that will ignite the imagination of developers and technical architects interested in Kafka and event-driven architectures. NewLimits, a shoe retail titan, suffers from inefficient data transfers due to outdated ETL pipelines and batch-based systems, frustrating internal teams and hampering development. Developers Ada and Jax explore Apache Kafka® and Confluent to develop real-time data streaming capabilities and build a proof-of-concept while fending off the Batch Gang's sabotage attempts.
In this course, Sophie Blee-Goldman, Apache Kafka® Committer and Software Engineer, gets you started with Kafka Streams. To understand Kafka Streams, you need to begin with Apache Kafka—a distributed, scalable, elastic, and fault-tolerant event-streaming platform. Kafka Streams is declarative, so you state what you want to do, rather than how to do it. Imagine that you have a topic, from which you'd like to filter all records marked with the color "red." You could accomplish this with plain Kafka, but the equivalent Kafka Streams code would only take a third of the lines.
By leveraging event streaming and vector embeddings, GPT-4 can be modified into real-time GenAI support agent that provides precise, context-aware responses. To build a real-world, production application with GPT-4, businesses need to integrate its general capabilities with their unique data:
Event Stream Integration: Merge data from various systems into a unified customer view for real-time access.
Search-Based Prompting: Enhance responses by adding relevant customer data to each prompt.
Vector Database Usage: Employ embeddings and a vector database for quick access to policy and knowledge base information.
Plugin Integration and Observability: Integrate ChatGPT plugins for streamlined interactions and capture conversation data for continuous performance improvement.
Flink SQL is a standards-compliant SQL engine for processing both batch and streaming data with the scalability, performance, and consistency of Apache Flink. This is a very expressive API, based on powerful abstractions, that can be used to quickly develop many common use cases. This video explains the relationship of Flink SQL to the Table and DataStream APIs. Through an extended example, it illustrates the stream/table duality at the heart of Flink SQL.
How to Use Flink SQL, Streamlit, and Kafka: Part 1
This part of the series focuses on how to make an app that allows a user to select a stock and view a live chart of the stock’s bid prices, calculated every five seconds. The source of the data is the Alpaca Market Data API. We’ll hook up a Kafka producer to the websocket stream and send data to a Kafka topic in Confluent Cloud. Then we’ll use Flink SQL within Confluent Cloud’s Flink SQL workspace to tumble an average bid price every five seconds. Finally, we’ll use a Kafka consumer to receive that data and populate it to a Streamlit component in real time.
Siemens & Brose: Pioneers in The Fourth Industrial Revolution
Industry 4.0, otherwise known as the Fourth Industrial Revolution, marks the normalisation of the smart factory. The foundation for this revolution lies in the convergence of the Internet of Things (IoT), cloud computing, analytics, and AI. Two manufacturing giants, Siemens and Brose, not only embraced this transformation but have harnessed the full potential of data streaming due to early adoption and advanced utilisation of data streaming technologies. Siemens made a strategic move to Confluent Cloud for cost-effectiveness, overcoming legacy system challenges and improving data update times. Brose, driven by its Future Pro Program, established a robust on-premise data streaming platform to manage a heterogeneous machine landscape.
Key Challenges of Digital Transformation in Manufacturing
Digital transformation (DX) is a comprehensive business strategy centered on leveraging digital technology. According to PTC, companies that succeed in DX, beat their ROI goals by an average of 50%, while those that fail, miss expectations by an average of 30%. Although DX challenges for manufacturing include strategy alignment, employee buy-in, and effective change management, data management and alignment is a key success factor for many advanced initiatives due to its magnitude: "Understanding the systems where data lives, the accessibility of that data, the connectedness of that data, the hygiene of that data are all necessary evaluations—and challenges."
“Generative AI’s potential in R&D is perhaps less well recognized than its potential in other business functions. Still, our research indicates the technology could deliver productivity with a value ranging from 10 to 15 percent of overall R&D costs.” Despite some limitations in applicability across industries, GenAI can enhance productivity by quickly producing candidate designs, reducing design costs, and optimising designs for manufacturing to lower costs associated with logistics and production. Other benefits of GenAI in R&D include the potential to develop higher-quality products as well as speed up testing and trial phases.
