Skip to main content

How to transfer data from Cloud Datastore to Big Query in Google Cloud Platform

If you are here I am assuming that you are looking to migrate the data from Cloud Datastore to Big Query because you want to do some analysis and are frustrated by limitations imposed by GQL (Google Query Language).

First of all you need to create a backup of the data in datastore. Use the Datastore Admin tool provided by Google to take a backup and store it automatically in the Cloud Storage bucket.

Select all the entities and press 'Backup Entities'. Give a backup name, select Google Cloud Storage as backup storage destination and specify a bucket name.

Once the backup job is completed, you will see the backup listed. You can select a backup and press 'Info' and see the details (Entities are masked in the screenshot below).

Go to the bucket mentioned in 'Handle' and you will see the file mentioned above. You will also see many more files with similar names, ending with .backup_info (e.g. ahRzfmpkYS1wZC1zbG8tc2FuZGJveHJBCxIcX0FFX0RhdGFzdG9yZUFkbWluX09wZXJhdGlvbhix_-4DDAsSFl9BRV9CYWNrdXBfSW5mb3JtYXRpb24YAQw.JobDetailsEntity.backup_info)

This is the backup file for a specific entity which you will need to specify when creating a table in Big Query.

Head over to Big Query and create a new dataset.

In Location field select 'Google Cloud Storage' and give the location of backup file for the specific entity. File format is 'Cloud Datastore Backup'.

Like the one we found earlier: gs://jda_so__78700310-e2f9-4cf2-8f20-dd325de09a4d_data_bkup/ahRzfmpkYS1wZC1zbG8tc2FuZGJveHJBCxIcX0FFX0RhdGFzdG9yZUFkbWluX09wZXJhdGlvbhix_-4DDAsSFl9BRV9CYWNrdXBfSW5mb3JtYXRpb24YAQw.JobDetailsEntity.backup_info.

Here the bucket name: jda_so__78700310-e2f9-4cf2-8f20-dd325de09a4d_data_bkup is coming from the 'Handle' field in the backup information in the datastore admin and the file name you got in the previous step!

Specify the name of the table you want to create in Big Query in 'Destination' field. Press 'Create Table' and if everything is correct, the job will complete successfully.  Select a table from the left panel and click on Preview to see the data populated. And you are done!

Let me know in comments if you have any questions.


Popular posts from this blog

File upload problem: UTF-8 encoding not honored when form has multipart/form-data

The problem that I was facing was something like this. I was using Apache Commons File Upload library to upload and download some file.

I had a form in which user can upload a file and another field 'name' in which she can give any name to the file being loaded.

When I submitted the form, the file was uploaded fine but the value in name field was garbled. I followed all the possible suggestions I found:

<%@page pageEncoding="UTF-8"%> set. <%@page contentType="text/html;charset=UTF-8"%gt; set after the first directive. <meta equiv="Content-Type" content="text/html;charset=UTF-8"> in the head. enctype="multipart/form-data" attribute in the form. accept-charset="UTF-8" attribute in the form.
in the Servlet:
before doing any operations on request object: request.setCharacterEncoding("UTF-8"); For accessing the value

FileItem item = (FileItem);

if (item.isFormField()) {

//For regular…

java.lang.IllegalArgumentException: Malformed \uxxxx encoding

I was getting this exception during build while running ant. Googling didn't help much and I was flummoxed because the same code was running fine till now.

My code reads a text file and does some operations on the basis of values read. It was only when I saw the text files I understood the error. I had copied the text in wordpad and saved it as .txt file. Wordpad had put lot of formatting information before and after the content. Also there was "\par" after every line, which was giving this error.

So moral of the story: if you get this exception check your properties file (or any other file that your code might be reading.)

Easiest way to print Timestamp in Java

Rather than using Calendar.getTime() we can use java.sql.Timestamp class to get the time stamp which gives date and time till millisecond precision.

System.out.println(new Timestamp(System.currentTimeMillis()));

Above will give you current timestamp in this format: 2010-07-27 16:37:45.39