null datetime field and json fixtures

[Malik Rumi]

There are a ton of answers to this question out there - if you can wade
through all the ones that refer to forms and not models. The consensus
seems to be that datetime has to be set to both blank=true and null=true.
And when I put in one test row of data manually through the admin, I could
leave my datetime field blank with no problems. But when I tried to load
the 89 other instances through a fixture, I got error after error.

As near as I can tell, if you leave a json value blank, loaddata says it
isn't a json document. But if you put the empty string, '', Null, None,
"Null", "None" or "0000-00-00" in there, it is rejected as not valid date
format. Therefore, it seems there is no way to do this through a fixture.
My question: Is this correct? If so, is there any alternative to doing it
all manually? Thanks.

[Avraham Serour]

Try not setting any value to the field, it should be saved with the default
value

Post by Malik Rumi
There are a ton of answers to this question out there - if you can wade
through all the ones that refer to forms and not models. The consensus
seems to be that datetime has to be set to both blank=true and null=true.
And when I put in one test row of data manually through the admin, I could
leave my datetime field blank with no problems. But when I tried to load
the 89 other instances through a fixture, I got error after error.
As near as I can tell, if you leave a json value blank, loaddata says it
isn't a json document. But if you put the empty string, '', Null, None,
"Null", "None" or "0000-00-00" in there, it is rejected as not valid date
format. Therefore, it seems there is no way to do this through a fixture.
My question: Is this correct? If so, is there any alternative to doing it
all manually? Thanks.
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/db367bcf-8cbf-4001-8976-f602e93576d5%40googlegroups.com
<https://groups.google.com/d/msgid/django-users/db367bcf-8cbf-4001-8976-f602e93576d5%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[James Schneider]

This post might be inappropriate. Click to display it.

[Malik Rumi]

First, thank you to both Avraham and James. In my view, it isn't said often
enough on the internet, and I've had many questions go without a response
at all. So please understand that I mean it very much when I say I deeply
appreciate the time you took to help me.

Now as to the matter at hand: The field in question is 'sunsetdate',
referring to the date something ends. It must be optional since most of my
objects/data don't, or haven't, ended. The first time I tried this I had
the empty string, '', in all rows in that column.

(cannon)***@Tetuoan2:~/Projects/cannon/jamf$ python manage.py
loaddata essell/fixtures/test22byhand.json

Traceback (most recent call last):

File
"/home/malikarumi/Projects/cannon/local/lib/python2.7/site-packages/django/core/serializers/python.py",
line 174, in Deserializer

raise base.DeserializationError.WithData(e, d['model'],
d.get('pk'), field_value)

django.core.serializers.base.DeserializationError: Problem installing
fixture
'/home/malikarumi/Projects/cannon/jamf/essell/fixtures/test22byhand.json':
[u"'' value has an invalid date format. It must be in YYYY-MM-DD format."]:
(essell.Code:pk=None) field_value was ''

I replaced “”, - the empty string - with ',', that is, nothing, no string,
no quote, nothing but the comma

File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode

raise ValueError("No JSON object could be decoded")

django.core.serializers.base.DeserializationError: Problem installing
fixture
'/home/malikarumi/Projects/cannon/jamf/essell/fixtures/test22byhand.json':
No JSON object could be decoded

So now I assume this is because there are no quotes on sunsetdate. Note how
that one error in that one field makes the whole document 'not json'.

Now I have to put the date in the way they want it, which is not what I
want but not the end of the world since I don't have to display this field
if there is no actual sunset date.

File
"/home/malikarumi/Projects/cannon/local/lib/python2.7/site-packages/django/core/serializers/python.py",
line 174, in Deserializer

raise base.DeserializationError.WithData(e, d['model'],
d.get('pk'), field_value)

django.core.serializers.base.DeserializationError: Problem installing
fixture
'/home/malikarumi/Projects/cannon/jamf/essell/fixtures/test22byhand.json':
[u"'null' value has an invalid date format. It must be in YYYY-MM-DD
format."]: (essell.Code:pk=None) field_value was 'null'

django.core.serializers.base.DeserializationError: Problem installing
fixture
'/home/malikarumi/Projects/cannon/jamf/essell/fixtures/test22byhand.json':
[u"'0000-00-00' value has the correct format (YYYY-MM-DD) but it is an
invalid date."]: (essell.Code:pk=None) field_value was '0000-00-00'

Last night before posting I hacked serializers/python.py to make an
exception for sunsetdate, and that apparently worked, but now I have this:

File
"/home/malikarumi/Projects/cannon/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py",
line 2390, in get_db_prep_value

value = uuid.UUID(value)

File "/usr/lib/python2.7/uuid.py", line 134, in __init__

raise ValueError('badly formed hexadecimal UUID string')

ValueError: Problem installing fixture
'/home/malikarumi/Projects/cannon/jamf/essell/fixtures/test22byhand.json':
badly formed hexadecimal UUID string

The only uuid currently in this fixture is the one I got from Django when I
put another model (only one row) in with a fixture (no date field) and it
worked. I needed the uuid of that object to put into the foreign key of the
model I am having trouble with now.

Finally, this morning after reading your responses I decided to try putting
in null with no comma, since I had already tried it with no quotes, and of
course that didn't work. But then in fooling with it I put null in right
next to the colon, with no spaces or quotes, followed by a comma, and this
time null turned blue in my code editor, which suggests it was working. But
then I got the 'badly formed hexadecimal UUID string error again.

In sum, there is a special syntax to using null which is not obvious, but
thanks for making me take another look and try again. The uuid error seems
to be something else. There is a bugfix on github from November of 2014, so
I don't (yet) know why I'm having this issue. I may post that later as a
separate question. Thanks again!

Post by James Schneider

Post by Malik Rumi
There are a ton of answers to this question out there - if you can wade
through all the ones that refer to forms and not models. The consensus
seems to be that datetime has to be set to both blank=true and null=true.
And when I put in one test row of data manually through the admin, I could
leave my datetime field blank with no problems. But when I tried to load
the 89 other instances through a fixture, I got error after error.

A blank form field is different than a missing field in a JSON dump, as
each is handled differently within the Django workflow. A programmatic way
to look at this would be: (blank=True) != (null=True).
blank=True controls whether or not a form will accept an empty value in
the form field as part of the validation mechanism for that field. It has
no bearing on model objects created through other means, such as part of a
fixture, or some view code that runs foo.save() somewhere.
Conversely, null=True only applies to models and saving to the database,
and even then, is not part of the validation mechanism when foo.save() is
called. It really just serves as a flag to the model inspection mechanism
for generating the schema that the database uses. Enforcement of missing
fields or fields with a Null/None value is done at the database level, not
at the model level within Python/Django. Obviously there are ways to inject
validation at the model level to prevent such issues, but very few, if any,
are used by default.
In form fields that accept arbitrary text input, a blank field is
submitted as an empty string. DateTime form fields are actually the same
under the hood, but most use some fancy JS to pretty it up with date-time
pop up selectors. If left blank in the form, an empty string is sent
through the form submission. Django then coerces the data (empty string)
for the DateTime form field to None for use later by the view/model, which
is not the case for other CharField-type fields that are left as an empty
https://docs.djangoproject.com/en/1.9/ref/forms/fields/#charfield
https://docs.djangoproject.com/en/1.9/ref/forms/fields/#datetimefield
When a blank (empty) value is provided through a form for a DateTime
*form* field, the DateTime *model* field will be populated with None, and
the resulting foo.save() operation will save the DateTime *model* field
as NULL in the database.
That's why for *model* CharField definitions (and other similar model
fields), it is a common pattern to only see blank=True, and NOT null=True,
because model CharFields are either saved with the text that was submitted,
or an empty string, but never as NULL (even with null=True).

Post by Malik Rumi
As near as I can tell, if you leave a json value blank, loaddata says it
isn't a json document. But if you put the empty string, '', Null, None,
"Null", "None" or "0000-00-00" in there, it is rejected as not valid date
format. Therefore, it seems there is no way to do this through a fixture.
My question: Is this correct? If so, is there any alternative to doing it
all manually? Thanks.

When you are talking fixtures, everything I just mentioned about forms is
thrown out (unless of course you are manually submitting the data in your
fixtures programmatically through Django forms, which is not entirely
uncommon since you can take advantage of form validation to eliminate/clean
bad data rows during an import). I only mentioned forms to help understand
why a 'blank' value in the Admin seems to work fine but your fixture is
failing.
The likely reason your fixture is failing is because you are expecting the
coercion of a missing/blank field to be handled like it is when data is
submitted through the form process. Django is expecting fixture data to be
pre-processed already from a known-good data source, and I believe that the
data is inserted using very low-level database API calls, and is not using
the standard ORM magic that we all know and love.
When you say you "leave a json value blank", what do you mean? Are you
removing the field entirely from the object definition? It appears that you
are keeping the field defined, but you want the equivalent of Python's
None. Python equates None to null (bare keyword in JSON), so your field
should probably be something like "updated_on: null," (without quotes). See
https://docs.python.org/3/library/json.html#py-to-json-table
Anyway, it sounds like the data source you are using is not correctly
serializing what is to become your problematic DateTime model fields. You
1. Fix the data source and have it generate/serialize to a proper
representation of your data, using either a real date or null to represent
an 'empty' or 'no date specified' value. An empty string does not make
sense as a value for a DateTime field (hence the coercion gymnastics that
form fields perform). Preferred.
2. Write a custom management command to manually de-serialize the data
and create it using the ORM and forms to take advantage of the Django
batteries. Not as graceful or efficient, but does grant you pretty explicit
control over how incoming data is filtered, cleaned, and inserted into the
database if you need to go that far. It probably isn't as bad as it sounds.
Most of my management commands are less than 10 lines, aside from imports,
but I'm not doing any fancy (de)serialization of incoming data, either.
I'm hoping that this entire email wasn't just a TL;DR; use null (without
quotes) to represent an empty datetime field in JSON...
It would also be helpful to see the errors and tracebacks that you are
getting when loading your fixtures. I could be completely wrong here,
especially if the error is something silly like your JSON file is missing a
comma or semi-colon.
-James
--

Post by Malik Rumi
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/db367bcf-8cbf-4001-8976-f602e93576d5%40googlegroups.com
<https://groups.google.com/d/msgid/django-users/db367bcf-8cbf-4001-8976-f602e93576d5%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "Django users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/django-users/Q4zybgExDyY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciW%3Ddm%3DM9Eh10Y8zcB%2Bvjx6E0UQC6a0mzsyyELmb2x2qzA%40mail.gmail.com
<https://groups.google.com/d/msgid/django-users/CA%2Be%2BciW%3Ddm%3DM9Eh10Y8zcB%2Bvjx6E0UQC6a0mzsyyELmb2x2qzA%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[James Schneider]

Post by Malik Rumi
Last night before posting I hacked serializers/python.py to make an

Yeah...don't do that. Fix your data.

Post by Malik Rumi
File
"/home/malikarumi/Projects/cannon/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py",
line 2390, in get_db_prep_value
value = uuid.UUID(value)
File "/usr/lib/python2.7/uuid.py", line 134, in __init__
raise ValueError('badly formed hexadecimal UUID string')
ValueError: Problem installing fixture
badly formed hexadecimal UUID string
The only uuid currently in this fixture is the one I got from Django when
I put another model (only one row) in with a fixture (no date field) and it
worked. I needed the uuid of that object to put into the foreign key of the
model I am having trouble with now.

The uuid.UUID() function is somewhat forgiving when it comes to providing
values. See https://docs.python.org/3.5/library/uuid.html. Does the UUID in
your JSON data match any of those formats? The only common format for a
UUID that I've seen that doesn't match any of those formats would be
'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' which is a string that contains
dashes, but no surrounding braces. I believe that's the format that is
pulled when using UUID's from Django installations by default. I'm actually
surprised the Python UUID library doesn't support it, but maybe it's one of
those RFC things that specifies the formats that must be accepted.

Post by Malik Rumi
Finally, this morning after reading your responses I decided to try
putting in null with no comma, since I had already tried it with no quotes,
and of course that didn't work. But then in fooling with it I put null in
right next to the colon, with no spaces or quotes, followed by a comma, and
this time null turned blue in my code editor, which suggests it was
working. But then I got the 'badly formed hexadecimal UUID string error
again.

JSON has rules for how data should be formatted. You may want to look those
up for guidelines.

Post by Malik Rumi
In sum, there is a special syntax to using null which is not obvious, but
thanks for making me take another look and try again. The uuid error seems
to be something else. There is a bugfix on github from November of 2014, so
I don't (yet) know why I'm having this issue. I may post that later as a
separate question. Thanks again!

Out of curiosity, did you generate the JSON manually, or did you use a
library (like the built-in JSON library in Python) to create the data dump
for you? From the sound of it, either you've mucked with it a bunch
yourself (which almost always leads to these types of issues with JSON), or
it was generated manually to look like JSON. Any proven serializer library
likely wouldn't generate data that resulted in the errors that you are
seeing (at least as far as the null issue is concerned). It would be
possible to get a 'malformed' UUID, since the serializer may not have been
configured correctly or aware of the intended format for the UUID values,
but would still have generated valid JSON.

-James

[Malik Rumi]

I am pursuing the debug option now to see what I can learn.

On Fri, Feb 19, 2016 at 5:29 PM, Michal Petrucha <

In [1]: import uuid
In [2]: uuid.UUID('61877565-5fe5-4175-9f2b-d24704df0b74')
Out[2]: UUID('61877565-5fe5-4175-9f2b-d24704df0b74')
BUT
In [3]: uuid.UUID4('61877565-5fe5-4175-9f2b-d24704df0b74')

---------------------------------------------------------------------------

AttributeError Traceback (most recent call

last)

<ipython-input-3-8d12e49c94ea> in <module>()
----> 1 uuid.UUID4('61877565-5fe5-4175-9f2b-d24704df0b74')
AttributeError: 'module' object has no attribute 'UUID4'
So uuid.UUID4() [or 3 or whatever] can only be used a certain way, as in
Michael's example?

There's no such thing as UUID4 in the Python uuid module -- there's
UUID (which is the class representing UUIDs), and there's the function
uuid4 (note that it's lowercase, it takes no arguments, and it will
return an instance of UUID).

The remaining question for me, then, is *what is the proper format *for a
uuid in a json fixture, or any other document I am trying to mass import
into Django?
61877565-5fe5-4175-9f2b-d24704df0b74 - (apparently not)
'61877565-5fe5-4175-9f2b-d24704df0b74'
('61877565-5fe5-4175-9f2b-d24704df0b74')
UUID('61877565-5fe5-4175-9f2b-d24704df0b74')
urn:uuid('61877565-5fe5-4175-9f2b-d24704df0b74')
some other variation I haven't come up with yet?
"48189959-be4c-4f10-819a-f1657061b3cd", "arrow": "Amendment II",
"shorttitle": "", "popularname": "Keep & Bear Arms", "acronym": "",
"offcite": "", "brokenarrow": "", "slug": "amendment-ii-keep-bear-arms",
"codetext": "A well regulated Militia, being necessary to the security

of a

free State, the right of the people to keep and bear Arms, shall not be
"''", "postdate": "2016-02-05 13:06:53.20548-06", "crossref": "''",

"Constitution",

"codelevelsortseq": "1", "siblingrank": "11", "childof_id": "",
"jurisdiction_id": "e6e11b06-ea3b-4e98-a31f-9a83447ad884"} }, {"model"
....
As you can see, the uuid is double quoted, but so are all the keys and
values. This is normal json format as I understand it. So should the uuid
be single quoted inside the double quotes? i.e.
"'61877565-5fe5-4175-9f2b-d24704df0b74'"
How do I get this done? Thanks.

The two UUIDs in that JSON snippet look correct at a first (and
second) glance. Have you tried using a debugger, or just a
quick-and-dirty print statement to see what value is being passed into
uuid.UUID that makes it raise an exception?
My guess would be that the JSON file contains a value somewhere that
is not a correct UUID, but I may be wrong, of course.
Regards,
Michal
--
You received this message because you are subscribed to a topic in the
Google Groups "Django users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/django-users/Q4zybgExDyY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/20160219232920.GF880%40koniiiik.org
.
For more options, visit https://groups.google.com/d/optout.

[James Schneider]

Post by Malik Rumi
I am pursuing the debug option now to see what I can learn.

Have you looked at enabling tracebacks (--traceback) and increasing the
verbosity of the loaddata command (-v {0,1,2,3}, --verbosity {0,1,2,3})?

$ python manage.py help loaddata
usage: manage.py loaddata [-h] [--version] [-v {0,1,2,3}]
[--settings SETTINGS] [--pythonpath PYTHONPATH]
[--traceback] [--no-color] [--database DATABASE]
[--app APP_LABEL] [--ignorenonexistent]
fixture [fixture ...]

Installs the named fixture(s) in the database.

positional arguments:
fixture Fixture labels.

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v {0,1,2,3}, --verbosity {0,1,2,3}
Verbosity level; 0=minimal output, 1=normal output,
2=verbose output, 3=very verbose output
--settings SETTINGS The Python path to a settings module, e.g.
"myproject.settings.main". If this isn't provided,
the
DJANGO_SETTINGS_MODULE environment variable will be
used.
--pythonpath PYTHONPATH
A directory to add to the Python path, e.g.
"/home/djangoprojects/myproject".
--traceback Raise on CommandError exceptions
--no-color Don't colorize the command output.
--database DATABASE Nominates a specific database to load fixtures into.
Defaults to the "default" database.
--app APP_LABEL Only look for fixtures in the specified app.
--ignorenonexistent, -i
Ignores entries in the serialized data for fields
that
do not currently exist on the model.


-James

[Malik Rumi]

This will probably be very helpful. Thanks.

Post by James Schneider

Post by Malik Rumi
I am pursuing the debug option now to see what I can learn.

Have you looked at enabling tracebacks (--traceback) and increasing the
verbosity of the loaddata command (-v {0,1,2,3}, --verbosity {0,1,2,3})?
$ python manage.py help loaddata
usage: manage.py loaddata [-h] [--version] [-v {0,1,2,3}]
[--settings SETTINGS] [--pythonpath PYTHONPATH]
[--traceback] [--no-color] [--database DATABASE]
[--app APP_LABEL] [--ignorenonexistent]
fixture [fixture ...]
Installs the named fixture(s) in the database.
fixture Fixture labels.
-h, --help show this help message and exit
--version show program's version number and exit
-v {0,1,2,3}, --verbosity {0,1,2,3}
Verbosity level; 0=minimal output, 1=normal output,
2=verbose output, 3=very verbose output
--settings SETTINGS The Python path to a settings module, e.g.
"myproject.settings.main". If this isn't provided,
the
DJANGO_SETTINGS_MODULE environment variable will be
used.
--pythonpath PYTHONPATH
A directory to add to the Python path, e.g.
"/home/djangoprojects/myproject".
--traceback Raise on CommandError exceptions
--no-color Don't colorize the command output.
--database DATABASE Nominates a specific database to load fixtures into.
Defaults to the "default" database.
--app APP_LABEL Only look for fixtures in the specified app.
--ignorenonexistent, -i
Ignores entries in the serialized data for fields
that
do not currently exist on the model.
-James
--
You received this message because you are subscribed to a topic in the
Google Groups "Django users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/django-users/Q4zybgExDyY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciVUtK1LtfpfsNBRF1chO_wxG_pabr8F3wZRZVR6G0SwqQ%40mail.gmail.com
<https://groups.google.com/d/msgid/django-users/CA%2Be%2BciVUtK1LtfpfsNBRF1chO_wxG_pabr8F3wZRZVR6G0SwqQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Malik Rumi]

Sorry to have dropped out for a few days there. James, when I said I only
have this one I should clarify. I had 90 records I wanted to put into one
model, and 1 record to put into another. The 1 record is linked to by a fk
from the other 90. For whatever reason, I was able to put the single record
fixture in without issue. Once I started having trouble with the others,
one of the many things I tried was taking all uuid's out to see if that
helped, and of course it didn't. So at one time the only uuid in the
fixture was the fk link to the model with only a single record in it. That
didn't work either.

All my pks at this moment are uuids. I read the SO link. I hope this does
not come across as too naive, but how would I check this as that person
did? Is it as simple as asking the shell what the uuid of x is, or is there
more to it because it might report back a uuid even though there is some
other 'ghost' integer floating around as well?

Post by Malik Rumi
The only uuid currently in this fixture is the one I got from Django when

Post by Malik Rumi
I put another model (only one row) in with a fixture (no date field) and it
worked. I needed the uuid of that object to put into the foreign key of the
model I am having trouble with now.

So I was sniffing around the Internet for ideas, and I came across this
http://stackoverflow.com/questions/32445546/django-uuidfield-modelfield-causes-error-in-django-admin-badly-formed-hexadecim
You mentioned that this is the only UUID in your fixture (which seems
strange). Do you have a mixture of UUID's and integer PK's? It's possible
that the UUID errors are actually resulting from an integer being passed to
uuid.UUID() rather than one of your real UUID's.
-James
--
You received this message because you are subscribed to a topic in the
Google Groups "Django users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/django-users/Q4zybgExDyY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciX%3Dz0KEvJJdMF7%3D3MKrWsuvhO6P%2B6%2BRso_QYM4Y6%3D4gsA%40mail.gmail.com
<https://groups.google.com/d/msgid/django-users/CA%2Be%2BciX%3Dz0KEvJJdMF7%3D3MKrWsuvhO6P%2B6%2BRso_QYM4Y6%3D4gsA%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.