Running Commentary

Google has recently given a new feature known as 'Dataflow Templates' which have got a number of tasks that you can do without writing any code.   Click the link on top of Dataflow page: You can select task from the list of available options: Select 'Bulk Delete Entities in Cloud Datastore' from the list.  Fill in the required parameters. Instead of table name, it asks for GQL query. For selecting all the entities in a table give "select * from " . IMPORTANT: In the optional parameters, you should specify the namespace/tenant for which you want to delete the entities.  Not specifying namespace will delete ALL the records from the selected table across the tenants .  Once done, press 'Run Job'. Dataflow will launch and do the cleanup. You can read documentation  here . Source code of all templates is also on  github .

There is a big chance that your kind (table) name is of the format . because there are various tables in your module. e.g. I had StoreService.Store and StoreService.Employee as two of the kinds in my service named StoreService. In Google Cloud Console, if you write a GQL to query to fetch/filter the entities you will get an error: SELECT * from StoreService.Store GQL query error:  Encountered "." at line 1, column 27. Was expecting one of: "group", "limit", "offset", "order", "where" Enclosing the Kind name in single quotes doesn't help either: SELECT * from 'StoreService.Store' GQL query error:  Encountered "'StoreService.Store'" at line 1, column 15. Was expecting one of: , To do it right, you need to enclose the name in backquotes (that key below ~ in the keyboard)!!! SELECT * from `StoreService.Store` This information is hidden in Datastore documentation here . Try finding i...

If you are here I am assuming that you are looking to migrate the data from Cloud Datastore to Big Query because you want to do some analysis and are frustrated by limitations imposed by GQL (Google Query Language). First of all you need to create a backup of the data in datastore. Use the Datastore Admin tool provided by Google to take a backup and store it automatically in the Cloud Storage bucket. Select all the entities and press 'Backup Entities'. Give a backup name, select Google Cloud Storage as backup storage destination and specify a bucket name. Once the backup job is completed, you will see the backup listed. You can select a backup and press 'Info' and see the details (Entities are masked in the screenshot below). Go to the bucket mentioned in 'Handle' and you will see the file mentioned above. You will also see many more files with similar names, ending with .backup_info (e.g. ahRzfmpkYS1wZC1zbG8tc2FuZGJveHJBCxIcX0FFX0RhdGFzdG9yZUFk...

Signed URLs are pretty nifty feature given by Google Cloud Platform to let anyone access your cloud storage (bucket or any file in the bucket) without need to sign in. Official documentation gives step by step details as to how to read/write to the bucket using gsutil or through a program. This article will tell you how to upload a file to the bucket using curl so that any client which doesn't have cloud SDK installed can do this using a simple script. This command creates a signed PUT URL for your bucket. gsutil signurl -c 'text/plain' -m PUT serviceAccount.json gs://test_bucket_location Here is my URL: https://storage.googleapis.com/test_sl?GoogleAccessId=my-project-id@appspot.gserviceaccount.com&Expires=1490266627&Signature=UfKBNHWtjLKSBEcUQUKDeQtSQV6YCleE9hGG%2BCxVEjDOmkDxwkC%2BPtEg63pjDBHyKhVOnhspP1%2FAVSr%2B%2Fty8Ps7MSQ0lM2YHkbPeqjTiUcAfsbdcuXUMbe3p8FysRUFMe2dSikehBJWtbYtjb%2BNCw3L09c7fLFyAoJafIcnoIz7iJGP%2Br6gAUkSnZXgbVjr6wjN%2FIaudXIqA...

You would already be aware that there are multiple options given by Google Cloud Platform to store data. Here is  Google documentation  on when to use which option: Google recommends using Google Cloud Storage (GCS) to store static content like files/videos etc. There is something called 'Blobstore' as well which is also used to store such content but it is on the way to being deprecated. This page talks about using GCS to store images. Look at  this page  to understand basic requirements for setup of GCS. In the Cloud Store Browser below following buckets are already available. If you select any bucket, you would be able to see the objects created in it.  Here you can see the image file in the 'jda-pd-slo-sandbox.appspot.com' bucket. You won't be able to add/delete files or folder from the browser if you don't have proper access but through code (running with the service account) it should not be a problem. Objects on GCS are immutable so you ca...