Driving Data Innovation: Progress and future plans for DATAMITE pilots
The DATAMITE project is steadily progressing through its six innovative pilots, each addressing unique data management challenges and demonstrating DATAMITE's potential for innovation in data sharing, governance and quality assessment. With milestones set through 2025, these pilots will strengthen the DATAMITE framework's ability to address diverse sectoral challenges and provide sustainable, scalable solutions for data sharing and management.
Here is an overview of the progress and upcoming milestones for each of these pilots:
Pilot 1: Data exchange between companies within the same corporate group
The infrastructure for data exchange is in place, with an on-premises DATAMITE ecosystem using PostgreSQL, MongoDB, and Cassandra. Test scenarios have been established, and a beta test API, developed using Django, facilitates interactions for OBREMO and FACSA use cases. Sensitive data management is in line with DATAMITE's legal advice, with problematic and controversial data being removed. Finally, cloud deployment testing is underway in collaboration with ITI.
Future steps
- Deployment of the DATAMITE framework, at least in the on-premises infrastructure.
- Communication and connection testing with API, with additional integration in case of interest.
- Collect and analyse initial KPIs.
- Produce multimedia materials to showcase test results.
- Drafting the pilot report.
Pilot 2: Corporate Multi-Site Data Exchange
The DATAMITE framework has been deployed in a development infrastructure. Test scenarios and synthetic datasets have been prepared in Month 23 (November 2024). The team is refining the data structures and running preliminary evaluations of the framework's performance.
Future steps
- Full deployment of the framework in the pilot development infrastructure.
- Finalise and execute use case validation.
- Set up production pilot infrastructure in Azure.
- Evaluate the DATAMITE framework on different operating systems, such as Red Hat Enterprise Linux flavours, to compare with traditional distributions such as Ubuntu or Debian.
- Submit a publication describing the pilot to the next AIAI 2025 conference.
Pilot 3: Offering Data to Service Providers with DataSpaces
Component deployment and testing began in month 17 (May) and is scheduled to be completed by month 25 (January 2025). The current priorities include deploying and testing the relevant DATAMITE components, finalizing the test scenarios, and contributing to the course design. Additionally, the deployment will soon be moved from Google Cloud VM to a permanent solution.
Future steps
- Carry out the first iteration of scenario testing between November 2024 and the end of January 2025, completing component deployment and testing, and gathering feedback to refine the second iteration of scenarios testing from March 2025.
- Migration of the deployment to an on-premises infrastructure.
- Deployment of Data Sharing components and publication of anonymised sensitive and composite data products.
- Draft the initial pilot report.
Pilot 4: Leverage Electricity Distribution Open Data
The internal infrastructure, which will be based on a MS Azure environment, is ready for deployment, while Data Quality, Data Governance and Data Support Tools have already started to be deployed. Testing will be divided into three levels and is focused on improving data collection and sharing processes.
Future steps
- Complete and validate pilot test scenarios.
- Deploy and test components from the Data Support Tools, Data Quality and Data Governance modules until the end of January 2025, which will be the focus of the first two levels of the test scenarios
- Draft the initial pilot report
Pilot 5: Connecting eDWIN to Data Markets
The team has completed the integration with Pontus-X for the agri-food sector. Meteorological data has been integrated with AIM and normalisation pipelines for pest and production (agrifood domain) datasets are operational.
Future steps
- Complete the first automated data workflow.
- Integrate eDWIN with DATAMITE data quality and data governance tools.
- Conduct test scenarios and complete demonstrators by the end of February 2025.
Pilot 6: Connecting MISTRAL to the EU AI-ON-Demand Platform
Resource allocation on the cloud is underway, along with the deployment of the development environment to test the DATAMITE framework. The team is also currently defining data quality checks and analysing data connectivity for external catalogues such as AI-on-Demand and the Mistral Open Data Catalogue.
Future steps
- Integration of the DATAMITE framework with the Mistal platform.
- Implementation of a Mistral connector to retrieve (meta)data in the DATAMITE framework.
- Dissemination of the data catalogue with data sharing to a local open data catalogue on the one hand, and to an external AI-on-demand platform on the other.
- Set up user-defined custom rules.
- Enable data quality checks with the Data Quality Module.
If you want to always stay updated about our project, follow us on LinkedIn, Twitter and Bluesky!
DATAMITE takes the stage at the first Gaia-X Hub Spain Data Space Summit
The first Gaia-X Hub Spain Data Space Summit ("Primera cumbre de espacios datos", as it is called in Spanish) took place in Madrid on 3 and 4 December. The most relevant international event on Data Spaces in Spanish brought together prominent leaders of national initiatives, government representatives and European project managers in the field of Data Spaces, Technology and Artificial Intelligence, with a special focus on use cases and ongoing projects, to share their knowledge and experiences in a dynamic and collaborative environment.
DATAMITE, as a European project focused on the monetisation of data and data spaces, coordinated by a Spanish entity (ITI), did not miss the event. Jordi Arjona Aroca, from ITI and technical coordinator of DATAMITE, presented the key aspects of the project and how the DATAMITE framework will revolutionise the data monetisation landscape in a session entitled 'Technology for data spaces' on 4 December.
If you want to always stay updated about our project, follow us on LinkedIn, Twitter and Bluesky!
DATAMITE plenary meeting in Aachen
From 26 to 28 November, the DATAMITE consortium met in Aachen for its second - and last - plenary meeting of 2024. As at the previous plenary meeting in Poznan, the agenda was divided into a first day of CodeCamp with the developers involved in the project, followed by two days with the whole consortium, where each work package had its moment in the spotlight with presentations and/or workshops.
The first day was spent at the CodeCamp, as a warm-up for the plenary sessions that would take place over the next two days. This edition was less about coding and more about practical activities and shaping the future of the DATAMITE framework. All DATAMITE design and coding partners were divided into parallel sessions, according to their needs, in order to get the most out of CodeCamp.
On 27 and 28 November the remaining partners who were not involved in the codecamp joined in. The DATAMITE consortium exchanged ideas, discussed the results of the review meeting and set the roadmap for the final year of the project. In contrast to previous plenary meetings, and despite the structure of the agenda being divided into work packages, the Aachen meeting was much more practical and focused on encouraging the participation of all partners in all work packages through more interactive workshops and sessions.
Some of the main topics discussed were the six pilot projects of the project and the future steps to be taken before the end of 2025. Exciting things to come next year!
In addition to three intense days of work, the DATAMITE consortium enjoyed a guided tour of the Aachen Christmas Market and a visit to the Demonstration Factory, which is a central component of the Smart Logistics Cluster on the RWTH Aachen campus, thanks to the colleagues from FIR RWTH Aachen University who hosted this plenary meeting.
If you want to always stay updated about our project, follow us on LinkedIn, Twitter and Bluesky!
DATAMITE presented at the E.DSO's Innovation and Research Committee
On 28 November 2024, the Innovation and Research Committee (IRCO) of the European Distribution System Operators (E.DSO) held an internal meeting focusing on 'Setting up national data sharing frameworks and platforms. This was an important date for DATAMITE’s partner, HEDNO, who took the opportunity to present the project. HEDNO delivered a 15-minute presentation titled 'Data Monetisation, Interoperability, Trade and Exchange', after which there was a 10-minute Q&A session during which attendees could share their thoughts on the project and learn more about DATAMITE's work.
E.DSO is the key interface between Europe’s Distribution System Operators (DSOs) and the European institutions. E.DSO gathers 35 leading electricity DSOs in 20 countries, including 2 national associations, cooperating to ensure the reliability of Europe’s electricity supply for consumers and enabling their active participation in our energy system.
If you want to always stay updated about our project, follow us on LinkedIn, Twitter and Bluesky!
DATAMITE at InfoCom World 2024
The 26th InfoCom World Conference was held in Athens on 12 November 2024 under the theme 'Digital Greece: Time to take the leap'. The annual meeting focused on the dynamic development of the country (Greece) through the adoption and use of new technologies, the challenges and opportunities that arise, and the steps that need to be taken by market players to achieve the 'digital leap'!
The thematic sections of the conference covered key topics such as: private networks, artificial intelligence, data centres, e-government, smart cities and digital projects, with an emphasis on the practical application and benefits of new technologies, bringing together key speakers from Greece and abroad. DATAMITE was present at InfoCom 2024 in two ways.
On the one hand, DATAMITE participated in the session 'Unlocking the Power of Data with DATAMITE Project: Innovative Approaches to Data Monetization and Interoperability'. The session featured two members of the consortium as speakers: Jordi Arjona Aroca from ITI and technical coordinator of DATAMITE, and Daniela Fuchs from the Centre for Social Innovation. They were joined by Sotiris Koussouris from Suite5 Data Intelligence Solutions and Haris Mouratidis from the University of Essex. The session was moderated by another member of the DATAMITE consortium, Christos-Antonios Gizelis from OTE Group of Companies.
On the other hand, DATAMITE was represented during the whole day of the event at the OTE booth, where the partners of the consortium communicated and disseminated the project to the attendees through a roll-up.
If you want to always stay updated about our project, follow us on LinkedIn, Twitter and Bluesky!
DATAMITE presents a paper at ICCBDC 2024
DATAMITE presented its groundbreaking research at the International Conference on Cloud Computing and Big Data (ICCBDC) 2024, held at Oxford University, UK, from 15 to 17 August. The conference provides an international platform for engineers and scientists to discuss advances and applications of cloud and big data, with sessions on data modelling, machine learning, image processing and information security. DATAMITE's paper introduced a new dimension of data quality called 'Purity', which assesses the relevance and importance of data sets in distributed networks, particularly in cloud computing environments. The presentation at ICCBDC underlined DATAMITE's commitment to pioneering data quality solutions.
The paper - called 'Purity: a New Dimension for Measuring Data Centralization Quality' - written by researchers from Tecnalia, partner of DATAMITE, and from Deusto Institute of Technology, emphasises 'Purity' as a crucial metric for assessing the relevance and significance of datasets within decentralised networks, particularly in cloud computing environments. This metric enables organisations to predict data quality issues before data merges, thus optimising cloud resources and facilitating data-driven strategies. The research underscores that early identification of potential quality concerns can streamline processes, enabling more efficient and accurate data handling.
The effectiveness of the purity dimension is validated through a mobility use case, Intelligent Transportation Systems (ITS), analysing four datasets from varied domains. The study assessed each dataset's importance within the network using centrality indicators—degree, betweenness, and closeness—. Key insights included:
- Degree Centrality: Measures direct connections a dataset has, indicating connectivity-based importance.
- Betweenness Centrality: assesses how often a dataset acts as an intermediary in communication between other datasets, highlighting its role in information transfer.
- Closeness Centrality: Evaluates the efficiency of communication by measuring how quickly a dataset can reach others.
The paper also introduced a comprehensive quality evaluation framework covering six dimensions: accuracy, completeness, timeliness, validity, uniqueness, and consistency. Through mathematical formulations, this framework supports predictive quality evaluations of merged datasets. Future research will refine the 'Purity' metric, enhancing its applicability across various domains and data environments.
As part of the outstanding work carried out by the researchers and their presentation at the ICCBDC, they were awarded the Excellent Oral Presentation Award at the conference.
If you want to always stay updated about our project, follow us on LinkedIn, Twitter and Bluesky!
DATAMITE attends the AI Data, Robotics Forum 2024
The second edition of the AI, Data and Robotics Forum (ADRF) took place on 4 and 5 November in Eindhoven, the Netherlands. Under the theme 'European Sovereignty in AI, Data and Robotics', the ADRF programme featured visionary keynotes and thought-provoking panels, as well as a series of parallel sessions on key industrial and societal issues. It also provided an opportunity for the AI, data and robotics community to highlight emerging issues and gather critical mass to drive them forward.
DATAMITE did not want to miss the opportunity to participate in the Forum. The project was a Silver Sponsor of ADRF 2024 and the project coordinator, Santiago Cáceres (ITI), travelled to Eindhoven to present DATAMITE at the poster exhibition. It was a unique opportunity to discover emerging trends, meet and network with other projects and initiatives that contribute to shaping a sustainable, effective and coherent European AI, data and robotics ecosystem.
If you want to always stay updated about our project, follow us on LinkedIn, Twitter and Bluesky!
DATAMITE at Enlit Europe 2024
Enlit Europe, one of the biggest EU energy events, was held from 22 to 24 October in Milan, Italy. The conference attracts 15,000 visitors, brings together more than 700 exhibiting companies, over 80 thematic sessions and more than 100 EU-funded projects, and features 500 speakers. Thanks to our energy use cases, led by our partners E-REDES and HEDNO, DATAMITE has been a perfect match for Enlit Europe 2024.
E-REDES and Hedno, another partner of the consortium, brought DATAMITE to Enlit Europe with a very nice booth located at the EU Projects Zone. The booth had a screen to project the video presentation of the project and an information area with different materials such as brochures, used by the partners present at the event to make DATAMITE known among the thousands of attendees.
On the second day of the conference, Ricardo Almeida Henriques (E.-REDES) had the opportunity to present DATAMITE in the session ‘Energy efficiency and optimisation with digital technologies’. During the session, he demonstrated how DATAMITE can contribute to the energy transition from two action points. On the one hand, by boosting data exchange between energy stakeholders, currently being tested in the E-REDES pilot project. On the other hand, by enabling the development of new system planning tools by improving data accessibility and sharing, which is the main objective of the HEDNO pilot project.
If you want to always stay updated about our project, follow us on Twitter and LinkedIn!
DATAMITE at OCX and eSAAM 2024
The Open Community Experience 2024 (OCX), hosted by the Eclipse Foundation -a DATAMITE partner- was held from 22 to 24 October in Mainz, Germany. OCX is an inclusive place for diverse open source communities to come together to learn, share, and network.
DATAMITE has been represented during the three days of the event thanks to the Eclipse booth, from which the consortium partners have shown different materials of the project, such as brochures and roll-ups, and have also explained to the attendees the main objectives and achievements reached so far by DATAMITE.
OCX organises several collocated events, including eSAAM. The 4th Eclipse Security, Artificial Intelligence, Architecture, and Modelling Conference on DATA SPACES, took place during the first day, Tuesday 22, bringing together industry experts and researchers working on innovative software and systems solutions for data spaces, specifically focusing on security and privacy, artificial intelligence and machine learning, systems and software architecture, modelling and related challenges.
DATAMITE participated at eSAAM in two ways. On the one had as a proud sponsor of the conference, and on the other hand with Publication presentations by two of the consortium members within the ‘architecture’ block:
- ‘Enabling EDIHs as Data Space intermediaries’, by Daniel Sáez Domingo from ITI. He explained the critical role of Hubs in making the Data Economy happen, in the sense of the importance to have trusted organisations like the eDIHs/i-Spaces providing enabling services to reduce the entry barrier to the data economy and particularly to the data spaces. Read the paper presented here.
- ‘Evolution of Extract-Transform-Load (ETL) processes towards data product pipelines’, by Gorka Zárate Martinez from TECNALIA. He talked of the emergence of data as a key asset, which has driven the development of infrastructures and tools to improve the ability of organisations to leverage data internally. Read the paper presented here.
If you want to always stay updated about our project, follow us on Twitter and LinkedIn!
Upcoming events
The DATAMITE consortium partners are constantly working to showcase the progress of the project, presenting the latest publications and articles at various national and international events. Here is a list of events not to be missed between November and December 2024.
ADRF 2024
The AI, Data, Robotics Forum (ADRF) will take place from 4 to 5 of November in Eindhoven, Netherlands. Under the theme ‘European Sovereignty in AI, Data and Robotics’ the ADR Forum is a premier annual event organised by the AI, Data and Robotics Association (Adra) in collaboration with the European Commission that brings together leading experts, innovators, policymakers, and enthusiasts from the AI, Data, and Robotics community. DATAMITE is a proud sponsor of this year’s edition, and the project will be represented through the DATAMITE coordinator, Santiago Cáceres (ITI), and the presentation of a project poster.
InfoCom World 2024
The 26th InfoCom World Conference will take place next 12th of November in Athens, Greece, under the title ‘Digital Greece; Time for a leap!’. The annual meeting of the market for Technology, Information Technology and Telecommunications, will focus on the dynamic development of the country through the adoption and exploitation of new technologies, the challenges and opportunities that arise, and the steps that must be taken by market stakeholders, to implement the digital leap. DATAMITE will be represented by one of its Greek partners, OTE, which will have a booth at the conference and will participate in the sessions organised during the day.
Más allá del Horizonte
The 28th of November will take place the 'Más allá del Horizonte' (Beyond the Horizon) Conference in Oviedo, Spain. The conference is organised by the Spanish Centre for Technological Development and Innovation (CDTI), in collaboration with the Spanish Ministry of Science, Innovation and Universities, the European Commission and the City of Oviedo. Its main objective is to analyse the functioning of Horizon Europe and its major initiatives so far, including projects funded by Horizon Europe with Spanish participation. DATAMITE will be one of the projects that can be seen in the poster exhibition. The participation of DATAMITE in the conference is done together with the project coordinator, ITI - Instituto Tecnológico de Informática de Valencia-.
If you want to always stay updated about our project, follow us on Twitter and LinkedIn!