This is regarding aws kinesis put-record command through AWS CLI.
I am able to input text data using kinesis cli.
aws kinesis put-record --cli-binary-format raw-in-base64-out --stream-name NagaTZeusTestStream --partition-key 1 --data 2 --region us-west-2
Here the data is 2
But how can i put a csv file in place of 2 as data.
And how can i put a csv file which is in s3
For example :
aws kinesis put-record --cli-binary-format raw-in-base64-out --stream-name NagaTZeusStream --partition-key 1 --data s3://cona-sample-salesforce-data/testdata/ --region us-west
In this case the file csv file in the s3 bucket should be uploaded as a data record , but kinesis is considering the s3 path itself as a data string.
Any help would be apprecieated. Thanks in advance
Kinesis Streams lets you write opaque blobs of data. The Kinesis PutRecord
API (which is what the AWS CLI kinesis put-record
command calls) expects you to give it a blob of data. If the data is stored in S3, it is your responsibility to load that data to send to Kinesis.
A common Kinesis pattern when working with "large" data is to put actual data into some other storage system (S3 being a great example) and then writing the "location" of that data (in this case, an S3 path) to Kinesis. With Kinesis Streams, your throughput (and costs) to/from Kinesis are directly affected by the amount of data you read/write. This of course requires coordination between the publisher and consumer, on the exact format (and semantics) of messages. If this is the case, you should look at what your consumers are expecting the format of your message to be.
But the moral of the story here is that Kinesis (and the CLI's put-record
) are going to put/write exactly what you give it.
Hi Adam. Thanks for responding.
Hi @nagasatishchilakamarti - if you find an answer useful, feel free to give it an upvote, but more importantly, if it successfully answered your question, be sure to mark it as the "Accepted" answer.