IssueTracker

VERSION:
release: "1.5.2"
timestamp: 1308730906
api_versions: ['1']

The bulkloader will not import models without properties. Example:

class MetaObject(db.Model):
    """
    Property-less object. Identified by application set key.
    """
    pass


In an application you can use these entities like this:

db.put([MetaObject(key_name=make_key(obj)) for obj in objs])
db.get([Key.from_path('MetaObject', make_key(obj)) for obj in objs])
db.delete([Key.from_path('MetaObject', make_key(obj)) for obj in objs])

Now the problem occurred when I tried to import data using the bulkloader. After looking through the bulkloader code, the bug turned out to be in the EncodeContent method (lines 1400-1406):

1365   def EncodeContent(self, rows, loader=None):
1366     """Encodes row data to the wire format.
1367
1368     Args:
1369       rows: A list of pairs of a line number and a list of column values.
1370       loader: Used for dependency injection.
1371
1372     Returns:
1373       A list of datastore.Entity instances.
1374
1375     Raises:
1376       ConfigurationError: if no loader is defined for self.kind
1377     """
1378     if not loader:
1379       try:
1380         loader = Loader.RegisteredLoader(self.kind)
1381       except KeyError:
1382         logger.error('No Loader defined for kind %s.' % self.kind)
1383         raise ConfigurationError('No Loader defined for kind %s.' % self.kind)
1384     entities = []
1385     for line_number, values in rows:
1386       key = loader.generate_key(line_number, values)
1387       if isinstance(key, datastore.Key):
1388         parent = key.parent()
1389         key = key.name()
1390       else:
1391         parent = None
1392       entity = loader.create_entity(values, key_name=key, parent=parent)
1393
1394       def ToEntity(entity):
1395         if isinstance(entity, db.Model):
1396           return entity._populate_entity()
1397         else:
1398           return entity
1399
1400       if not entity:
1401
1402         continue
1403       if isinstance(entity, list):
1404         entities.extend(map(ToEntity, entity))
1405       elif entity:
1406         entities.append(ToEntity(entity))
1407
1408     return entities

Because (will also post an issue for this one) the datastore Entity object subclasses dict without overriding the __nonzero__ or __len__ methods an Entity that does not contain any properties, but does have a key, will not *be* True (makes "if not entity" true even whena key has been set) and will thus not be appended to entities.

Here is a diff that fixes this in the bulkloader OR by overriding __nonzero__ in Entity (either one works):

--- bulkloader.py       2011-08-27 18:21:36.000000000 +0200
+++ bulkloader_fixed.py 2011-08-27 18:22:48.000000000 +0200
@@ -1397,12 +1397,9 @@
         else:
           return entity

-      if not entity:
-
-        continue
       if isinstance(entity, list):
         entities.extend(map(ToEntity, entity))
-      elif entity:
+      else:
         entities.append(ToEntity(entity))

     return entities
--- datastore.py        2011-08-27 18:41:16.000000000 +0200
+++ datastore_fixed.py  2011-08-27 18:40:50.000000000 +0200
@@ -644,6 +644,12 @@

     self.__key = Key._FromPb(ref)

+  def __nonzero__(self):
+      if len(self):
+          return True
+      if self.__key:
+          return True
+
   def app(self):
     """Returns the name of the application that created this entity, a
     string or None if not set.