BigQuery on python. upload_from_file job not done #3343
Description
I am running a test to upload a .json file into an bigquery table and i cant get the job done.
Here is my code :
def stream_data2(dataset_name, table_name, json_data):
bigquery_client = bigquery.Client()
dataset = bigquery_client.dataset(dataset_name)
table = dataset.table(table_name)
# Reload the table to get the schema.
table.reload()
job = table.upload_from_file(
json_data, source_format='NEWLINE_DELIMITED_JSON',
encoding='UTF-8',
create_disposition='CREATE_IF_NEEDED',
write_disposition='WRITE_APPEND')
pprint.pprint(job)
with open('clarin.json', 'rb') as json_data:
stream_data2(DATASET_NAME, TABLE_NAME, json_data)The job ended in fail with this error:
> Errors:
> file-00000000: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. (error code: invalid)
> file-00000000: JSON parsing error in row starting at position 0: . No such field: custom. (error code: invalid)
>
This has no sense becouse i create the table with a json file with the same structure.
Any suggestion?
Thanks
Activity
dhermes commentedon Apr 28, 2017
@tswast Is JSON even supported as an upload type? (
source_format='NEWLINE_DELIMITED_JSON'seems to indicate yes)tswast commentedon Apr 28, 2017
Yes, JSON is supported. Keep in mind that it's "Newline Delimited". That means each row not table is a separate JSON object, each separated by a newline.
patitonav commentedon Apr 28, 2017
tswast commentedon Apr 28, 2017
I recommend using the
bqcommand line tool to experiment with the file format & schema. Once you have a file that BigQuery accepts, then you can automate it with this library. https://cloud.google.com/bigquery/bq-command-line-tooldhermes commentedon Apr 28, 2017
@patitonav I am going to pre-emptively close this, since it seems your issue is with your data formatting and this isn't really an issue with this library.
Let me know if you'd like to re-open this issue.
Thanks for the assist @tswast, helpful as always!
chore(deps): bump urllib3 from 2.0.7 to 2.2.2 in /.kokoro (#12803)
samples: add samples from pubsub/cloud-client (#134)
Update dependency google-cloud-pubsub to v1.4.2 in Storage and Pub/Su…
samples: add cloud client samples from python-docs-samples (#626)