IssueTracker

Proposal: make backup/restore a "first class citizen" of App Engine.

Backup/restore should be an integral part of all systems. App Engine protects live data very well with multiple replicas across multiple data centers. However, developer and user errors can still lead to an unrecoverable loss of data.

In App Engine, we should be able to specify our backup policies in a backup.yaml file in our root directory. See below for an example. Incremental backups can be  supported using a similar under-the-hood technique that the HRD migration tool uses to perform incremental migrations. Google Storage "folders" should be possible using filename prefixes. Automatic deletion of old backups should be supported. Backups should be able to include specific Kinds and/or Namespaces, or exclude specific Kinds and/or Namespaces. I think Google Storage should be the only required storage location because (1) it is the most cost effective offering from Google, and (2) it has good tooling/interop.

When restoring, it is important to be able to include/exclude particular Kinds and/or include/exclude particular namespaces. You should be able to provide a Kind and/or Namespace mapping (e.g., suffix) so that Kinds/Namespaces can be restored to an alternate location (as opposed to fully overwriting live data). This allows the client to write code to do a more surgical restore of data (e.g., restore a particular account).

Sample backup.yaml:

backups:

- name: Daily Important
  type: incremental                     # only add new data, except on a new snapshot
  schedule: every day of month 01:00    # run nightly at 1.00a
  snapshot_every: 7                     # roll to a new snapshot every 7th backup
  harvest_age: 90                       # delete backups older than 90 days
  bucket: my-backups                    # backup to this bucket on Google Storage
  queue: backup                         # use this queue (and rate)
  prefix: daily-important/              # use this file prefix on Google Storage (which is a 'folder' in this case)
  exclude_kinds:                        # do _not_ backup these Kinds
    - Audit
    - VisitorLog
   
- name: Hourly Suspicious Audit
  type: incremental                     # only add new data
  schedule: every 1 hour synchronized   # run hourly, on the hour
  harvest_age: 2                        # delete backups older than 2 days
  bucket: suspicious                    # backup to this bucket on Google Storage
  queue: backup                         # use this queue (and rate)
  prefix: to-investigate/               # use this file prefix on Google Storage
  namespaces:                           # only backup these namespaces, could alternatively specify exclude_namespaces
    - suspicious-activity
  kinds:                                # only backup these Kinds
    - Audit
    - VisitorLog
   
- name: Monthly Everything
  type: full                            # every backup creates a full set of data
  schedule: 1st day of month 02:00      # run on the first
  harvest_age: 750                      # keep two years worth of monthly backups
  bucket: compliance                    # backup to this bucket on Google Storate
  queue: backup                         # use this queue (and rate)