Warm tip: This article is reproduced from stackoverflow.com, please click
jobs mysql schema talend

How to Guess schema in Mysqlinput on the fly in Talend

发布于 2020-04-21 10:58:09

I've build a job that copy data from a mysql db table to b mysql table. The table columns are the same except sometimes a new column can be added in table a db.

i want to retrieve all the columns from a to b but only those that exists in table b. i was able to put in the query specific select colume statment that exists in table b like:

select coulmn1,column2,columns3... from table a

the issue is if i add a new column in b that matches a the talend job schema in Mysqlinput should be changed as well cause i work with build in type.

Is there a way to force the schema columns during the job running?

Questioner
Itay Regev
Viewed
15
Ibrahim Mezouar 2018-01-18 01:52

If you are using a subscription version of Talend, you can use the dynamic column type. You can define a single column for your input of type "Dynamic" and map it to a column of the same type in your output component. This will dynamically get columns from table a and map them to the same columns in table b. Here's an example.
If you are using Talend Open Studio, things get a little trickier as Talend expects a list of columns for the input and output components that need to be defined at design time.

Here's a solution I put together to work around this limitation.

enter image description here

The idea is to list all table a's columns that are present in table b. Then convert it to a comma separated list of columns, in my example id,Theme,name and store it in a global variable COLUMN_LIST. A second output of the tMap builds the same list of columns, but this time putting single quotes between columns (so as they can be used as parameters to the CONCAT function later), then add single quotes to the beginning and end, like so: "'", id,"','",Theme,"','",name,"'" and store it in a global variable CONCAT_LIST.

On the next subjob, I query table a using the CONCAT function, giving it the list of columns to be concatenated CONCAT_LIST, thus retrieving each record in a single column like so 'value1', 'value2',..etc

Then at last I execute an INSERT query against table b, by specifying the list of columns given by the global variable COLUMN_LIST, and the values to be inserted as a single string resulting from the CONCAT function (row6.values).

This solution is generic, if you replace your table names by context variables, you can use it to copy data from any MySQL table to another table.