DTP: The Future of Data Transfer?

Most people today have data, documents, images and more tied up in a multitude of data stores. Getting everything from Point A to Point B should be relatively easy, but it's not. Suppose, for example, that a user wants to move photos from Google to Facebook. The process should be as easy as clicking a few buttons, but it’s actually pretty complicated. For companies that have made major forays into the ever-changing environs of social media and other platforms, the hassles involved with downloading data--and the dangers of exposing that data along the way--are not trivial.  

Industry heavy-hitters want to change that with the Data Transfer Project, an ambitious undertaking that, if successful, will create an open standard for data portability. Spearheaded by Google, Facebook, Twitter and Microsoft, the Data Transfer Project wants to make it easier for users to transfer data from one service to another without having to download and re-upload the data. According to the group's mission statement, making it easier “to choose among services facilitates competition, empowers individuals to try new services, and enables them to choose the offering that best suits their needs.” 

Facebook put it this way: “Moving your data between any two services can be complicated because every service is built differently and uses different types of data that may require unique privacy controls and settings. For example, you might use an app where you share photos publicly, a social networking app where you share updates with friends, and a fitness app for tracking your workouts. People increasingly want to be able to move their data among different kinds of services like these, but they expect that the companies that help them do that will also protect their data. These are the kinds of issues the Data Transfer Project will tackle.”

As for the timing of the project (announced earlier this year), Google software engineer Brian Willard cited several reasons in a podcast. “Mobile-only usage is growing, and Internet usage is growing in areas where connectivity is metered or connectivity isn’t that great,” he said. “In both of those cases, downloading data isn’t that tenable a solution.”

Willard went on to explain that while Google already lets users transfer Google data to other sources like Box or OneDrive, “you’re still left with a zip file in the cloud. DTP tries to solve all of those problems by transferring your data right in the cloud directly to another service provider.”

Willard also mentioned Europe’s General Data Protection Regulation (GDPR) as an incentive. GDPR, which was implemented in May 2018, requires stringent data protection and privacy controls for all European citizens, and one of its requirements is focused on data portability. More specifically, it requires that citizens be able to “receive personal data they have provided to a controller in a structured, commonly used and machine readable format” and request that a controller transmits this data directly to another controller.

The Evolution of DTP

DTP's roots go back about a decade ago, when a group of Google engineers formed the Data Liberation Front. The team had the same idea—full data portability—but the timing wasn’t right yet. The next iteration of the idea came in 2011 with the launch of Takeout, a way for Google users to export data from one Google service to another. For example, a user could choose to export and download data from Google Contact, Tasks, Calendar or any other Google product in a variety of formats.

It was a good start, but Google engineers were dreaming bigger. While the ability to download data was a great start, they wanted a way to enable service-to-service transfer. The Data Transfer Project group was formed in 2017 to create an open-source, service-to-service data portability platform enabling data transfer between and among online service providers.

DTP in Action 

So how will it work? It’s complicated, Willard says, because each type of data—contacts, profile data, photos, and so on—has a different format. Further, the format for even one type of data, such as contacts, may be different depending on the service. To ensure smooth transfer, each company is responsible for building adapters to translate their APIs into a common format. Because the format is open, any company can build adapters to its APIs to gain access to information.

When the adapters are built, someone using a new photo sharing app, for example, might see a message asking whether he or she would like to import Facebook photos. The user would click on the link and, after being authenticated by the service, initiate the transfer.

While developing the project, Willard said, all participants are focusing heavily on data privacy and security. All data is encrypted both in transit and at rest with a unique key, and credentials are destroyed as soon as the transfer concludes. In addition, services must agree to allow data transfer among them, and will require that users authenticate each account independently. DTP also will monitor the number and frequency of transfers for each user and revoke tokens if they are leaked.

Once the system is up and running—which Willard says will take more time—there are many ways both individuals and companies can benefit. Individuals, for example, could use it to easily try out new services using their own data, or to back up their data to cloud storage.

Willard noted that there are additional options for advanced users. Because it’s an open source product, anyone can inspect the code to verify that data isn't being collected or used for profiling purposes. Tech-savvy users can also download and run an instance of the framework themselves. A small company looking to grow and acquire users could build an adapter and gain access to customer data if those customers give permission, among many other applications of the technology. 

Next Steps 

Willard hopes to attract more major companies into the DTP fold, something that is critical to expanding its adapter base. So far, the project has adapters for Google, Microsoft, Twitter, Instagram, Flickr, Remember the Milk and SmugMug, and can handle five data types: mail, contacts, calendar, photos and tasks. In the meantime, Google plans to work on building its adapters into the Google user interface.

There are other decisions yet to be made, as well, such as how the system will be governed. For example, will there be an “adapter store”? How will the group allow or disallow adapters? Do they even have the right to disallow them, since it’s an open platform?

Over time, Willard hopes, DTP will fulfill its promise as an open platform for data portability while protecting users’ data and keeping them in control.

The Data Transfer Project’s open source code can be found at datatransferproject.dev.