AWS Summit Zurich 2025 - Real-time Energy Trading: Alpiq's Strategic Data Platform on AWS

Initial Situation and Challenges [00:10]

• Alpik initially experienced panic when asked to share their data platform experience, feeling constantly busy with ongoing IT support and modernization.
• They identified a critical issue of managing 10,000 applications for 1,300 employees, leading to an unsustainable workload for their 150 IT employees.
• The company processed 3 billion transactions daily, resulting in nearly 2 petabytes of data storage from 30 different sources, which consumed 20% of the IT budget.
• Key challenges were categorized into data (silos, quality, governance, source of truth), data engineering (infrastructure, fragmented tech stack, scalability), and business team issues (lack of self-service, SLA breaches).
• These challenges resulted in a centralized data team acting as a bottleneck, demanding impossible breadth of knowledge from its members, frequent troubleshooting, inconsistent outputs, and no room for innovation.

"In a nutshell, we were always in firefighting mode."

Embracing Data Mesh Principles [08:21]

• Alpik decided on a fundamental revolution encompassing culture, people, technology, and process to tackle their data challenges.
• They adopted data mesh principles as the backbone of their data platform, aiming to overcome their previous limitations.
• The four core principles implemented were:
  • Domain Ownership: Shifting from a central bottleneck to each domain owning its data with clear boundaries.
  • Data as a Product: Treating data not as raw tables but as designed, maintained, and delivered products with a consumer mindset.
  • Self-Service Platform: Creating a domain-agnostic backbone that provides consistent capabilities for innovation without central IT dependency.
  • Federated Governance: Implementing policies and standards that apply universally but are automated and enforced efficiently.

"This principle were not just theory for us. They were the exact answers to our data engineering and business challenges and data mesh became the backbone of our data platform."

Platform Evolution and Architecture [09:43]

• The data platform evolved through stages: onboarding, marketplace & governance, and consumption infrastructure & operations, with data teams supporting high-value use cases.
• The architecture starts with diverse sources (ERP, operational databases, IoT, external APIs), flows into an ingestion layer supporting CDC, batch, and streaming, and then to the self-service platform core for quality, access control, metadata, and processing (SQL, Spark, streaming).
• Data is stored flexibly and reliably, then made consumable via dashboards, APIs, AI models, and custom applications, with the core principle of being repeatable, self-service, and secure.
• Technologies utilized include AWS native services (S3, EKS) for scalability and automation, complemented by Snowflake for analytics, DBT for transformation, and tools like Quicksite and Streamlit for data closer to business users, all tied together with CI/CD and monitoring tools like DataDog.

"The important point here is not about choosing AWS or external tools. It's about combining them in a smart way."

Operational Model and Developer Journey [14:20]

• The success of the platform hinges on how teams operate around the technology, emphasizing people and processes working together.
• The operational model includes:
  • Data Products Teams: Business domain-specific teams owning and producing data products.
  • Platform Team: Providing infrastructure, automation, and tools.
  • Governance Team: Defining and enforcing standards.
  • Consumer Teams: Utilizing trusted data to deliver business value.
• For developers, the journey involves CI/CD pipelines configured with YAML files for consistency and automation, utilizing reusable components for tasks like creating tables or managing Kafka topics.
• Data lands in a data lake (S3 and Iceberg) where it becomes structured and governed data products, discoverable via a data catalog (data.world) and accessible through query engines like Athena and Snowflake.

"Technology is only half of the story. What really makes this work is how teams operate around it because technology alone doesn't deliver value."

Business Case 1: Battery Energy Storage Systems (BESS) - Day One [18:16]

• This case addressed Alpik's role in power production and securing energy supply, focusing on Battery Energy Storage Systems (BESS).
• The objective was to gather market and grid data (capacities, prices, volumes) from facilities in Finland via REST APIs (from FINDS and Nord Pool) and internal systems, and provide standardized output.
• An ELT (Extract, Load, Transform) approach was used: data was extracted, loaded into Amazon S3 as a persistent staging area, and then transformed within the platform for historization, versioning, and standardization.
• The data product offered consumption via API, direct querying through Iceberg tables, and BI reporting tools like Quicksite.
• The implementation leveraged AWS services like EKS, PI Iceberg, and Amazon S3, with transformations managed within the platform, and data catalog updated by AWS Glue crawlers.

"The key message here is that raw external market data is turned into created trusted program products. Traders can then rely on this data for their decision making processes."

Business Case 2: Energy Trading and BESS - Day Two [24:17]

• This case focused on Alpik's energy trading activity, introducing real-time battery data into the system for faster trading decisions.
• The objective was to build a streaming modality, combining market/grid data with data from batteries (charged/not charged status) for energy traders.
• The architecture involved ingesting data through an internet gateway and load balancer to Fargate, then streaming it via Amazon MSK (managed Kafka).
• Apache Flink was used as the stream processor for real-time ETL, aggregation, and enrichment.
• Data flowed to a separate Kafka topic for internal services and to an S3 bucket for curated data, accessible via Athena for analytics.
• This case highlighted the use of reusable components, particularly for Kafka and Flink, demonstrating a shift from batch to real-time processing with low latency and high throughput.

"Fundamental streaming analytics shift our data processing paradigm from batch to real time processing and analyzing."

Business Case 3: Sales, Grid Data, and AI [32:14]

• This case involved ingesting data from Spanish transmission system operators to build a grid map, focusing on the challenge of frequently changing data formats provided by vendors.
• The solution aimed to automate schema change detection and update semantic models and visualizations, reducing manual effort and improving dashboarding for business users.
• Leveraging Amazon Q in Quicksite, the system automatically detects schema changes, calls models (via LightOn) to interpret these changes, updates the semantic model, and reuploads the updated version into the visualization system.
• A two-account architecture was used to provide flexibility for using different AI models from various providers via an LLM gateway on the Generative AI platform.
• End users can now interact with complex grid data and pricing through natural language queries, with Q finding the right datasets and relationships to provide visualizations.

"This solution is far from finished... But what is very relevant of this architecture is actually that it is again down with reusable components."

Conclusions and Impact [38:03]

• The data platform yielded significant advantages, including a 60% reduction in time to delivery due to smaller teams, reusable components, and modern technologies.
• Infrastructure costs were reduced by an estimated 75% through better data quality controls, governance, and duplicate prevention.
• For the grid model use case, reusable components automated data dictionary updates and provided user-friendly dashboarding options.
• The data platform successfully broke down data silos by introducing data products, and addressed data quality, governance, and unique source of truth issues.
• Scalability and fragmentation challenges were tackled through the modalities explained, and self-service/agility improved via the data mesh approach and AI.

"Considering where we started, our journey cannot be short. But given the results that we have obtained so far, the data platform is showing to be the right way forward."