Chapter 8 Anonymity and Confidentiality

Or, Statistical Disclosure Control, Masking, and De-Identification

8.1 Introduction

The confidentiality of many data that are ripe for analysis are covered under some form of legal, ethical, and moral constraints. How do we as researchers ensure that the data we work with–whether it’s in aggregated form or released as a micro (individual) record that is stripped of personal identifiers such as name–remains anonymous, thereby ensuring that any personal information remains private?

Many of the practical applications of statistical disclosure control can be traced back to the work of various statistical agencies around the world, who are confronted with the challenge of balancing increased expectations for openness and transparency (which translates into an increasing demand for the release of data) with “an increasing public consciousness about the privacy of individuals” (Bethlehem, 2009, p.342).

Statistical Disclosure Control (SDC) or Statistical Disclosure Limitation (SDL) seeks to protect statistical data in such a way that they can be released without giving away confidential information that can be linked to specific individuals or entities. (Hundepool et al., 2012, p.1)

Because statistical agencies have this role and take the responsibility very seriously (their reputations rest heavily on the willingness of individuals, businesses, and other public agencies to provide data to them) means much of the material on the topic comes directly or indirectly from those agencies (see, for example, the Australian National Statistical Service and Statistics Canada publications listed below).

A related topic is data masking, “the process of hiding original data with random characters or data.”

The main reason for applying masking to a data field is to protect data that is classified as personal identifiable data, personal sensitive data or commercially sensitive data, however the data must remain usable for the purposes of undertaking valid test cycles. It must also look real and appear consistent. Data masking page at Wikipedia

Sidebar: The Mosaic Effect

Often bandied about is the term “the mosaic effect”. A good definition comes from Vivek Kundra, who at the time was the U.S. Government’s Chief Information Officer:

Individual pieces of data when released independently may not reveal sensitive information but when combined, this “mosaic effect” could be used to derive personal information or information vital to national security. (“Testimony Resolving the Shroud of Secrecy”, 2010-03-23)

Further reading: “Sorry, your data can still be identified even if it’s anonymized”

Aggregation, masking, and other SDC methods should reduce (if not entirely eliminate) a “mosaic effect”. More research into this topic is required.

8.2 Laws, Ethics, and Morals

The Canadian Panel on Research Ethics had created a policy document that includes a chapter on Privacy and Confidentiality, covering the full range of concerns about the use of individual data.

8.3 Discussion

Wikipedia: Data anonymization

Some definitions: “What is the difference between”de-identified“,”anonymous“, and”coded" data?"

Justin Brickell and Vitaly Shmatikov, 2008: “The Cost of Privacy: Destruction of Data-Mining Utility in Anonymized Data Publishing”

Dary Hsu, “Techniques to Anonymize Human Data”, 2015

Gregory J. Matthews, Pétala Gardênia da Silva Estrela Tuy, and Robert K. Arthur, “An examination of statistical disclosure issues related to publication of aggregate statistics in the presence of a known subset of the dataset using Baseball Hall of Fame ballots”, Journal of Quantitative Analysis in Sports, 2017

Arvind Narayanan and Vitaly Shmatikov, 2008: “Robust De-anonymization of Large Datasets (How to Break Anonymity of the Netflix Prize Dataset)”

Arvind Narayanan and Vitaly Shmatikov, 2008: “Robust De-anonymization of Large Sparse Datasets”

Latanya Sweeney, “Information Explosion” in Confidentiality, Disclosure, and Data Access: Theory and Practical Applications for Statistical Agencies (Sweeney (2001))

Latanya Sweeney, Michael von Loewenfeldt, and Melissa Perry, “Saying it’s Anonymous Doesn’t Make It So: Re-identifications of “anonymized” law school data" in Journal of Technology Science (Latanya Sweeney (2018))

  • additional publications on the topic of data anonymization and re-identification by Latanya Sweeney can be found as part of her CV at

Pete Warden, “Why you can’t really anonymize your data”, 2011

8.4 Statistical agencies: principles and practices

8.4.1 Australia

National Statistical Service (Australia), “How to confidentialise data: the basic principles”

National Statistical Service (Australia), “Managing the risk of disclosure in the release of microdata”

8.4.2 Canada

Statistics Canada, “Disclosure control” (part of Statistics Canada Quality Guidelines, Catalogue 12-539-X)

Statistics Canada, 2011: A New Approach for the Development of a Public Use Microdata File for Canada’s 2011 National Household Survey, Catalogue no. 99-137-XWE2015001.

Mark Stinner, 2017, “Disclosure Control and Random Tabular Adjustment”, SSC Annual Meeting, June 2017. {PDF}

8.4.3 United Kingdom

Government Statistical Service (United Kingdom), 2009:National Statistician’s Guidance: Confidentiality of Official Statistics

Government Statistical Service (United Kingdom): “Statistical Disclosure Control”

Office of National Statistics (United Kingdom), 2004: National Statistics Code of Practice: Protocol on Data Access and Confidentiality

8.4.4 United States of America

Bureau of Labour Statistics (BLS) Disclosure Review Board (U.S.A.), 2012: Balancing Confidentiality Requirements with Data Users’ Information Needs

Census Bureau: Statistical Disclosure Control (SDC)

  • “Links and conventional references to research sponsored by the U.S. Census Bureau in the areas of statistical disclosure control, confidentiality, and disclosurelimitation.”

John Abowd, Tweetorial: Reconstruction-abetted re-identification attacks and other traditional vulnerabilities {unrolled twitter thread}

8.5 Other resources

8.5.1 Health agencies: “de-identification”

U.S. Department of Health & Human Sciences, Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule

Committee on Strategies for Responsible Sharing of Clinical Trial Data; Board on Health Sciences Policy; Institute of Medicine. Washington (DC): National Academies Press (US); 2015 Apr 20. Sharing Clinical Trial Data: Maximizing Benefits, Minimizing Risk: Appendix B: Concepts and Methods for De-identifying Clinical Trial Data

8.5.2 others

Information and Privacy Commissioner of Ontario, June 2016, De-identification Guidelines for Structured Data

Simson L. Garfinkel, October 2015, De-Identification of Personal Information, NISTIR 8053, National Institute of Standards and Technology.

Jelke Bethlehem, 2009: Applied Survey Methods: A Statistical Perspective, Wiley (Wiley Series in Survey Methodology)

  • The 13th and final chapter of this comprehensive outline of survey research methods is titled “Statistical Disclosure Control”. While the book focusses on data collected through sample surveys, the principles apply equally to census surveys and administrative records.

Children’s Hospital of Eastern Ontario Research Institute, 2007: Pan-Canadian De-Identification Guidelines for Personal Health Information

Josep Domingo-Ferrer, 2014, “Data Anonymization: A Tutorial”

Anco Hundepool, Josep Domingo-Ferrer, Luisa Franconi, Sarah Giessing, Eric Schulte Nordholt, Keith Spicer, Peter-Paul de Wolf, 2012: Statistical Disclosure Control, Wiley (Wiley Series in Survey Methodology)

  • Essentially a handbook of methods to ensure the privacy of individuals, it covers both microdata (individual records) and tabular data, recognizing the inherent differences in the risks associated with each.

Leon Willenborg and Ton de Wall, 1996: Statistical Disclosure Control in Practice, Springer (Lecture Notes in Statistics 111).

Sharon Hewitt, “Making Test Data Realistic – Without Taking It from Production” (

8.6 Specific methods

8.6.1 cell perturbation

Jean-Louis Tambay, June 2017: “A layered perturbation method for the protection of tabular outputs”, Survey Methodology (12-001-X), Statistics Canada, vol. 43, no. 1.

Gwenda Thompson, Stephen Broadfoot, Daniel Elazar, 2013: “Methodology for the Automatic Confidentialisation of Statistical Outputs from Remote Servers at the Australian Bureau of Statistics”, Joint UNECE/Eurostat work session on statistical data confidentiality, Ottawa, Canada, 28-30 October 2013.

8.6.2 k-anonymity

wikipedia: k-anonymity

Latanya Sweeney, “k-Anonymity: A Model for Protecting Privacy”, International Journal on Uncertainty, Fuzziness and Knowledge-based Systems, 10 (5), 2002; 557-570. 2002

Arvind Narayanan and Vitaly Shmatikov, “Robust De-anonymization of Large Sparse Datasets”

Roberto J. Bayardo and Rakesh Agrawal, “Data Privacy Through Optimal k-Anonymization”

Khaled El Emam and Fida Kamal Dankar, “Protecting Privacy Using k-Anonymity”, J Am Med Inform Assoc. 2008 Sep-Oct; 15(5): 627–637.

8.6.3 pq-Rule

Meghan O’Malley, Lawrence R. Ernst, “Practical Considerations in Applying the pq-Rule for Primary Disclosure Suppressions”, December 2007.

8.7 R

John Mount (1012) “Modeling Trick: Masked Variables”

8.7.1 {sdcMicro}

{sdcMicro} is the best available SDC tool I have found in the R space, so I’ve given it top-billing rather than my usual alpabetical listing.


CRAN page: sdcMicro: Statistical Disclosure Control Methods for Anonymization of Microdata and Risk Estimation

github page: sdcMicro (note: this is a sub-page for for a broader set of SDC tools)


Statistical Disclosure Control (SDCMicro) at International Household Survey Network.

Matthias Templ, “Statistical Disclosure Control for Microdata Using the R-Package sdcMicro”, Transactions on Data Privacy 1 (2008) 67–85.

Matthias Templ, Bernhard Meindl and Alexander Kowarik, Introduction to Statistical Disclosure Control (SDC), 2015/2017

Matthias Templ, Alexander Kowarik, Bernhard Meindl (2015) “Statistical Disclosure Control for Micro-Data Using the R Package sdcMicro”, Journal of Statistical Software, Vol.67 No.4

Alexander Kowarik and Matthias Templ, “Make Your Data Confidential with the sdcMicro and sdcMicroGUI packages”, 2012

Daniel Abril, Guillermo Navarro-Arribas and Vicenç Torra (2015) “Data Privacy with R”,, 2015-07-10.

8.7.2 {sdcMicroGUI}


CRAN page: sdcMicroGUI: Graphical User Interface for Package ‘sdcMicro’


Matthias Templ, Bernhard Meindl and Alexander Kowarik, Tutorial for sdcMicroGUI (and sdcMicro), 2015

8.7.5 {digest}


CRAN page: digest: Create Compact Hash Digests of R Objects


Jan Górecki (2014), “Data anonymization in R” * alternate source

8.7.6 {obfuscateR}

(Note: this package has not been submitted to CRAN, and is clearly in development/stalled)

github page: PedramNavid/obfuscateR


Latanya Sweeney, Melissa Perry, Michael von Loewenfeldt. 2018. “Saying It’s Anonymous Doesn’t Make It so: Re-Identifications of ‘Anonymized’ Law School Data.” Technology Science, November.

Sweeney, Latanya. 2001. “Information Explosion.” In Confidentiality, Disclosure, and Data Access: Theory and Practical Applications for Statistical Agencies, edited by J. Theeuwes L. Zayatz P. Doyle, 26. Washington, D.C.: Urban Institute.