I wanted to replicate Mysql tables held in GCP Compute Engine to the GC BigQuery. I referred this document : https://cloud.google.com/data-fusion/docs/tutorials/replicating-data/mysql-to-bigquery. so I Decided to use GCP Data Fusion for the Job.
Everything works fine and the Data is replicated in Bigquery. So I was testing different datatype support for this Replication.
Where I come up with issue in this Replication Pipeline, So whenever I try to put the 'DATE' datatype Column for the Data fusion replication, the whole table (Which contain 'DATE' Column) doesn't show up in BigQuery
It Creates the table with schema same as source and 'Date' datatype also present in Bigquery, and I have use the same Date format as supported by BigQuery.
I also gone through Data fusion logs, It shows pipeline is Loading the data perfectly fine into BigQuery, Also catches the new rows added into Mysql Table from source Mysql DB with inserts and updates as well. But somehow rows are not getting into Bigquery.
Did anyone used Data fusion Replication with 'Date' column Datatype ? Is this issue with BigQuery or Data Fusion ? Do I need to provide any manual setting in the BigQuery ? Can anyone please provide inputs on this ?
I'll mark this issue as Resolve. Since the Problem is with Data Fusion, and the latest version 6.4.1 now supports Datatime datatype while replicating in BigQuery. I'm Receiving correct date & datetime data now.
Thank you for all the Help :)