Public and private roles in managing data (Part 2: Data sharing)

Public and private roles in managing data (Part 2: Data sharing)

Phuah Eng Chye (9 May 2020)

“This digital Europe should reflect the best of Europe – open, fair, diverse, democratic, and confident. The EU can become a leading role model for a society empowered by data to make better decisions – in business and the public sector. To fulfil this ambition, the EU can build on a strong legal framework – in terms of data protection, fundamental rights, safety and cybersecurity – and its internal market with competitive companies of all sizes and varied industrial base…Moreover, it will have to improve its governance structures for handling data and to increase its pools of quality data available for use and reuse. Ultimately, Europe aims to capture the benefits of better use of data, including greater productivity and competitive markets, but also improvements in health and well-being, environment, transparent governance and convenient public services.

European Commission (2020) Communication: A European strategy for data.

Europe led the way in formalising a framework to uphold the primacy of individual privacy rights with its General Data Privacy Regulation (GDPR). This was followed by the unveiling of a strategy that recognises data “constitute a potential source of growth and innovation that should be tapped…And that data should be available to all – whether public or private, big or small, start-up or giant. This will help society to get the most out of innovation and competition and ensure that everyone benefits from a digital dividend”[1].

The recent strategy emphasising data use seems at odds with earlier-launched GDPR emphasis on privacy. The EC acknowledges geopolitical pressures were a major factor influencing its data strategies. Europe lags despite possessing “the technology, the know-how and a highly skilled workforce”. “Competitors such as China and the US are already innovating quickly and projecting their concepts of data access and use across the globe”. This is partly due to the different approaches. “In the US, the organisation of the data space is left to the private sector, with considerable concentration effects. China has a combination of government surveillance with a strong control of Big Tech companies over massive amounts of data without sufficient safeguards for individuals”.

Europe has chosen a middle path. “In order to release Europe’s potential, we have to find our European way, balancing the flow and wide use of data, while preserving high privacy, security, safety and ethical standards”. “The aim is to create a single European data space – a genuine single market for data, open to data from across the world – where personal as well as non-personal data, including sensitive business data, are secure and businesses also have easy access to an almost infinite amount of high-quality industrial data, boosting growth and creating value, while minimising the human carbon and environmental footprint. It should be a space where EU law can be enforced effectively, and where all data-driven products and services comply with the relevant norms of the EU’s single market. To this end, the EU should combine fit-for-purpose legislation and governance to ensure availability of data, with investments in standards, tools and infrastructures as well as competences for handling data. This favourable context, promoting incentives and choice, will lead to more data being stored and processed in the EU. The European data space will give businesses in the EU the possibility to build on the scale of the Single market. Common European rules and efficient enforcement mechanisms should ensure that data can flow within the EU and across sectors; European rules and values, in particular personal data protection, consumer protection legislation and competition law, are fully respected; the rules for access to and use of data are fair, practical and clear, and there are clear and trustworthy data governance mechanisms in place; there is an open, but assertive approach to international data flows, based on European values”.

The tension between the individual right to privacy and the growth benefits from data transparency is unavoidable. In this context, privacy limits the number of users and this reduces the quality and value of data. For data to generate aggregate value and growth, it needs to be visible and usable. The higher the number of users, the higher the quality and value of data. Data sharing is thus a key aspect of the transparency paradigm[2]. The EU’s middle path thus involves adopting a dual mandate that attempts to balance the conflicting goals of privacy and transparency. This is a common regulatory challenge.

The European Commission (EC) report outlined its strategies to facilitate data-sharing by segments:

  • From the public to the private sector. “Opening up government-held information is a long-standing EU policy. This data has been produced with public money and should therefore benefit society”. The aim is to “ensures that the public sector makes more of the data it produces easily available for use, in particular by SMEs but also for civil society, and the scientific community, in the framework of independent public policy evaluations.”
  • Sharing of data among the private sector. The EC acknowledges “data sharing between companies has not taken off at sufficient scale…due to a lack of economic incentives…lack of trust…imbalances in negotiating power, the fear of misappropriation of the data by third parties, and a lack of legal clarity”.
  • From the private to the public sector. “There is currently not enough private sector data available for use by the public sector to improve evidence-driven policy-making and public services such as mobility management or enhancing the scope and timeliness of official statistics, and hence their relevance in the context of new societal developments”.
  • Sharing of data among public authorities. “Sharing of data between public authorities…can make a considerable contribution to improving policy making and public services, but also to reduce the administrative burden on companies operating in the Single Market (once only principle)”.

The data strategy also highlighted the market imbalances in relation to data access and use and the over-dependence on external providers (such as for cloud services). The EC will “provide more guidance to stakeholders on the compliance of data sharing and pooling arrangements with EU competition law…In the exercise of its merger control powers, the Commission will look closely at the possible effects on competition of large-scale data accumulation through acquisitions and at the utility of data-access or data-sharing remedies to resolve any concerns…the Commission will consider how best to address more systemic issues related to platforms and data, including by ex-ante regulation if appropriate, to ensure that markets stay open and fair”.

The EC “intends to fund the establishment of EU-wide common, interoperable data spaces in strategic sectors. Such spaces aim at overcoming legal and technical barriers to data sharing across organisations, by combining the necessary tools and infrastructures and addressing issues of trust, for example by way of common rules developed for the space. The spaces will include: (i) the deployment of data-sharing tools and platforms; (ii) the creation of data governance frameworks; (iii) improving the availability, quality and interoperability of data – both in domain-specific settings and across sectors. Funding will also support authorities in the Member States in making high value data sets available for reuse in the different common data spaces. The support for data spaces will also cover data processing and computing capacities that comply with essential requirements in terms of environmental performance, security, data protection, interoperability and scalability”. The proposed nine common European data spaces are industrial (manufacturing), Green Deal, mobility, health, financial, energy, agriculture, public administration [to improve transparency and accountability of public spending and spending quality, fighting corruption, to address law enforcement needs and support the effective application of EU law and enable innovative gov tech, reg tech and legal tech applications supporting practitioners as well as other services of public interest] and skills”.

In relation to this, a High-Level Expert Group on Business-to-Government Data Sharing notes the potential benefits of data remains untapped “not only because the vast majority of data is in the hands of the private sector, but also because the public sector does not seem ready to realise the full potential of data. Due to organisational, technical and legal obstacles (as well as an overall lack of a data-sharing culture) business-to-government (B2G) data-sharing partnerships are still largely isolated, short-term collaborations”. The expert group therefore recommend establishing national governance structures that support B2G data sharing; promoting data steward functions, introducing pilots for B2G data-sharing collaborations, harmonising B2G data-sharing processes, establishing mechanisms for accountability, transparency and compliance with ethical principles, raising public awareness on the societal benefits of data sharing and fostering a data literate public sector.

Gabriella Cattaneo, Giorgio Micheletti and Cristina Pepato points out the key issues affecting the evolution of the data governance model are “how the ownership, access, control and exploitation of data assets will be managed. To put it more bluntly: who will have power on data and what will governments do about it?” Their study highlighted two extreme scenarios: “On the one hand, a data governance model where a few data holders (private or public) control most of data assets; on the other hand, an open and participatory data governance model, based on sharing and transparency”. Hence, the debate over possible risks and the social challenges posed by the potential abuse of data requires “a balanced model of data governance ensuring the respect of human rights, but also the exploitation of opportunities and the fair distribution of benefits”.

However, they note “implementations’ hurdles remain, especially concerning the tension between extracting value added from data but still respecting GDPR principles…We foresee the GDPR to create gradually a successful harmonisation of regulation across the EU, but we suspect that there will be a need for adjustments and revisions. Removing barriers to the flow of non-personal data across Europe is a critical success factor to unlock the exploitation of European datasets at a scale and scope sufficient for the new data-driven processes such as machine learning”.

Gabriella Cattaneo, Giorgio Micheletti and Cristina Pepato suggest “there is room for policy intervention”. This includes raising “awareness about the advantages offered by data monetisation” and creating “a trusted environment around data monetisation… user-friendly data exchange mechanisms, confidentiality, licensing and pricing agreements and, not least, clarity on data access rights, help build trust between data holders and data re-users”. In addition, there is a need to “improve clarity on the legal framework affecting data monetisation….a lack of legal clarity regarding data ownership rights and/or about what can be lawfully done with datasets (usage of data), or again the difficulty in understanding/meeting the legal requirements on data protection in B2B transactions as the main reasons for not fully engaging in data monetisation exchanges. As companies cherish contractual freedom and direct bilateral agreements, awareness-raising measures, voluntary schemes, non-legislative interventions and enhanced regulatory guidance are likely to produce beneficial effects. Governments should also “increase funding for SMEs to engage in data monetisation…SMEs in particular could greatly benefit from targeted financial aid aimed at scale up data sharing technical solutions or help companies to invest in marketing solutions and communication activities to increase awareness among potential data users”. Hence, “more research on data monetisation mechanisms at play in other parts of the world, especially in the United States, Japan but also in China and India, could be beneficial to help the European data economy grasp new opportunities related to data sharing and monetisation”.

Analysis by the European Political Strategy Centre suggests “an alternative route could be considered, in which the regulatory framework would – while respecting the framework of the GDPR regime – maximise accessibility to data by any entity capable of generating value from it”. Similar to the way open government facilitates the release of government data to the public, “open data regimes should be extended to the private domain, to the degree that allowing access does not undermine the investment made by the data services that first processed the dataset”. An example is mobility data where “anonymised aggregated samples could be shared with authorities or researchers to develop applications capable of monitoring and preventing the evolution of pandemic threats”.

“Public intervention could prompt data-sharing, while also ensuring a limited scope for liability in the case of data-sharing for public interest. Access conditions would be designed such that the (often small) marginal costs for data sharing are covered, while incentives to invest in data collection are preserved”. “Increasing competition at the data service level could transfer value to data generators and data users and thus emerges as a more promising avenue to address issues concerning the fair distribution of value across the value chain than the definition of ownership rights”. “Competition can be stimulated through the design of a regulatory framework that promotes interoperability of platforms, for example through open application programme interfaces (APIs)…Likewise, competition can be stimulated through portability, i.e. by ensuring that data is easily transferable from one service to another by their users. This means enhancing transparency in data processing and favouring the adoption of common standards, for example for data formatting so that different services can easily read or map the same dataset on their own platforms. These principles are already part of the regime governing personal data under the GDPR. Data portability and interoperability are even more important in the field of machine-generated data, and accompanying rules are urgently needed so as not to lower services’ incentives to invest in data-driven innovation…Limits should be envisaged whenever data transfers would entail undermining a service’s business model”.

There are challenges to data-sharing. The Australian Government Productivity Commission Inquiry report highlights “private sector data owners are leading the way in finding innovative uses for data. Governments across Australia also hold lots of data, but are typically not using it to the extent that opportunities being taken overseas exemplify, and lack a comprehensive plan to do so in most cases”.

The Australian report highlights the main barrier to sharing data is not privacy regulation. The reluctance to release data – “be it because of inherent risk aversion or prior bad experiences” –  within government agencies “which focuses on avoiding mistakes or embarrassment and achieving consensus rather than seizing opportunities…officials and politicians will also be considering how information might be spun by the media, their opponents or those with direct commercial interests or an axe to grind”. In addition, there are concerns “for how an agency’s information will be used by other entities (loss of control), concerns about the cost of changing systems and processes to enable sharing of data, and concerns about exposure to criticism and/or legal risk”. The reluctance “is compounded by a paucity of positive incentives for agencies to share and release their data holdings…implications on additional processing involved and potential impact on funding”.

In addition, there are considerable misunderstandings and “difficulty in complying with law that varies across jurisdictions, managing inconsistencies in data collection practices and coordinating permissions across multiple and diverse data custodians…There is no overarching legislation…that addresses, in a whole-of-government way, how data is made available and used…the detrimental effect that multiple and inconsistent privacy laws have on sharing and the need for simplification: For governments to deliver services in the 21st century, they need to see themselves as the ‘government sector’ and not isolated eco-systems…Many pieces of legislation are drafted with explicit barriers to data sharing due to perceived risks, or have contradictory or overlapping data sharing restrictions…The list of legislation, international treaties and agreements, national strategies and policy statements…is extensive. Policy and regulation development is struggling to keep pace with government administrative changes, let alone the consequences of the development of digital technologies, increased user expectations, and increasing volumes of data. Navigating between differing interpretations and applications can lead to suboptimal outcomes and impose considerable overhead and often inertia”.

The report notes “custodians must also ensure that data is only released in accordance with all relevant legislation. In some cases, breach of legislative requirements can result in imprisonment or substantial fines for the data custodian personally. In light of this, it is unsurprising that data custodians would be risk averse. Internal procedures, which in some cases have evolved to take on a more substantial role than that originally intended in the legislation, can also have a chilling effect on data release”. In this regard, “custodians have created complex and inefficient approval processes for data access…involve multiple data owners, custodians and stewards, integration units, ethics committees and other advisory bodies…If the dataset contains information from multiple sources, approval usually needs to be sought from each data custodian separately and often sequentially…data custodians are often under-resourced in data provision, which only serves to compound delays in processing applications”. Committees can “have an important role, but create further duplication of effort”. Different committees are “dealing with different ethical aspects is not helpful in streamlining data access approval processes. Researchers still must submit applications to each separate committee and negotiate with each separately”.

The report highlights “data sharing between government agencies generally proceeds via a patchwork of mechanisms…a piecemeal process with a mix of different types of data sharing agreements including memoranda of understanding (MOUs), contracts, deeds, letters of exchange, undertakings, licences, head of agency/ministerial agreements, and public interest certificates. Arrangements might be one-off or ongoing, may or may not involve a payment for costs, are typically long and complex and involve negotiation… (MoU) agreement…requires multiple legal departments to be engaged on projects, along with external legal counsel. These are often drafted by policy and legal teams with little or any knowledge of where the data was captured, or what is the end-to-end journey of the data across a service change. Therefore, these MoUs are often complex, difficult to understand, unrealistically constrained to where the data can come from, or be used for, and bear little relation to what data is really required”.

In addition, government data is plagued by quality problems. For example, “data contained in legacy systems poses challenges such as: inability to link data; standardisation issues between data systems; gaps in metadata availability; and inconsistent storage formats – all of which can reduce data quality and cause difficulty in automating data provision…Fragmented data collection leads to the wasteful repetition of surveys and compliance-based reporting, thus placing an unnecessarily large regulatory burden on individuals, services, suppliers and businesses. In some cases, it may result in those parties submitting poor-quality survey data to save time and effort. The lack of coordination of data collection, management and publication standards across jurisdictions can lead to different measurements, making it difficult to aggregate data at a national level, or to share and link data across jurisdictions. Similarly, comparisons between jurisdictions become less effective – or even impossible – when datasets are incongruent”.

“The complexities of data storage and management are exacerbated by government agency restructuring (often after a change of government). Machinery of government (MOG) changes sometimes require agencies to transfer responsibilities for data collection, storage and custodianship. This can disrupt projects that are underway, including sharing arrangements…Moreover, where all data functions and responsibilities are not passed on, MOG changes can result in single datasets having differing collectors and custodians, or the data custodian being lost altogether”.

The report also note that “information provision to government from service providers is piecemeal…Sometimes government service providers are required under contract to provide information, yet contracting practices vary widely…situations where services have been privatised and the service provider has not been required under the contract to provide information to that government, which would enable more robust performance assessment and could subsequently be useful for policy development or infrastructure planning purposes. It is still rare for government-funded service providers to report information to the community in a comprehensive way, other than through annual reporting requirements. Even where data is provided to government, it is invariably not developed into data series for the public to use”.

In addition, intellectual property rights, use licenses and uncertainty over copyright status of data also impede the sharing or release of data. For example, “where public sector agencies use data generated by other organisations, including those from the private sector, the sharing and release of this data can be restricted by laws relating to copyright, patent, confidential information, trade secret and trademark”. “Particular issues arise where governments contract private sector organisations to provide services – such as public transport – and are unable to access data relating to the operation of those services”.

The report argues “marginal changes to existing structures and legislation will not suffice” to overcome the “lack of trust by both data custodians and users in existing data access processes and protections and numerous hurdles to sharing and releasing data are choking the use and value of Australia’s data”. It recommends reforms “aimed at moving from a system based on risk aversion and avoidance, to one based on transparency and confidence in data processes, treating data as an asset and not a threat. Significant change is needed for Australia’s open government agenda and the rights of consumers to data to catch up with achievements in competing economies”. The recommended reforms include “a new Data Sharing and Release Act, and a National Data Custodian to guide and monitor new access and use arrangements, including proactively managing risks and broader ethical considerations around data use”; “A new Comprehensive Right for consumers would give individuals and small/medium businesses opportunities for active use of their own data”; “the creation of a data sharing and release structure that indicates to all data custodians a strong and clear cultural shift towards better data use that can be dialled up for the sharing or release of higher-risk datasets”; and new arrangements under the Data Sharing and Release Act for national interest datasets.

In this regard, “streamlining governance arrangements – simplifying application processes, clarifying the role of data custodians and promoting mutual recognition between ethics committees – would go a long way towards supporting increased use of identifiable data…With this perspective, the principal challenges of policy are to smooth out some of the points of friction for: privacy protections; data licensing; data linking and sharing; data skills development and the agility of public consultation and development processes”.

Overall, governments around the world struggle to maintain a balance between privacy and transparency to get the best out of both worlds. Many countries and smart cities[3] are exploring various approaches to accommodate the dual mandate; in particular the shaping of information regulation to create safe harbours to protect government agencies and private firms from the legal risks of sharing data. In tandem with the changing environment and regulation, public and private roles in managing personal data will continue to evolve.


Australian Government Productivity Commission Inquiry Report (31 March 2017) “Data Availability and Use Inquiry Report”.

European Commission (19 February 2020) “Communication: A European strategy for data”.

European Commission (2020) “Towards a European strategy on business-to-government data sharing for the public interest”. Final report prepared by the High-Level Expert Group on Business-to-Government Data Sharing. file:///C:/Users/user/Downloads/B2GDataSharingExpertGroupReport.pdf

European Political Strategy Centre (11 January 2017) “Enter the data economy: EU policies for a thriving data ecosystem”. EPSC Strategic Notes Issue 21.

Gabriella Cattaneo, Giorgio Micheletti, Cristina Pepato (12 June 2019) “Second report on policy conclusions”. IDC, The Lisbon Council.

Phuah Eng Chye (12 October 2019) “Information and organisation: Shades of surveillance”.

Phuah Eng Chye (26 October 2019) “Information and organisation: Cross border data flows and spying”.

Phuah Eng Chye (21 December 2019) “The debate on regulating surveillance”.

Phuah Eng Chye (4 January 2020) “The economics and regulation of privacy”.

Phuah Eng Chye (18 January 2020) “Big data and the future for privacy”.

Phuah Eng Chye (15 February 2020) “The costs of privacy regulation”

Phuah Eng Chye (29 February 2020) “The journey from privacy to transparency (and back again)”.

Phuah Eng Chye (14 March 2020) “Features of transparency”.

Phuah Eng Chye (28 March 2020) “The transparency paradigm”.

Phuah Eng Chye (11 April 2020) “Anonymity, opacity and zones”.

Phuah Eng Chye (25 April 2020) “Public and private roles in managing data (Part 1: Surveillance)”.

[1] See European Commission “Communication: A European strategy for data”.

[2] See “The transparency paradigm”.

[3] See “Information and organisation: Cross border data flows and spying” and “The debate on regulating surveillance”.