I'm trying to stream rds through kinesis data stream but it is giving me this error:
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the PutRecord operation: 1 validation error detected: Value 'arn:aws:kinesis:us-west-2:xxxxxxxxxx:stream/rds-temp-leads-stream' at 'streamName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+
What can I do to fix this?
import json
import boto3
from datetime import datetime
from pymysqlreplication import BinLogStreamReader
from pymysqlreplication.row_event import (
DeleteRowsEvent,
UpdateRowsEvent,
WriteRowsEvent,
)
class DateTimeEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, datetime):
return o.isoformat()
return json.JSONEncoder.default(self, o)
def main():
mysql = {
"host": "",
"port":,
"user": "",
"passwd": "",
"db": ""}
kinesis = boto3.client("kinesis", region_name = 'us-west-2')
stream = BinLogStreamReader(
connection_settings = mysql,
server_id=100,
blocking = True,
log_file='mysql-bin.000003',
resume_stream=True,
only_events=[DeleteRowsEvent, WriteRowsEvent, UpdateRowsEvent])
for binlogevent in stream:
for row in binlogevent.rows:
print row
event = {"schema": binlogevent.schema,
"table": binlogevent.table,
"type": type(binlogevent).__name__,
"row": row
}
kinesis.put_record(StreamName="jhgjh", Data=json.dumps(event, cls=DateTimeEncoder), PartitionKey="default")
#print json.dumps(event)
if __name__ == "__main__":
main()
remove 'arn:aws:kinesis:us-west-2:xxxxxxxxxx:stream/rds-temp-leads-stream' this from stream name. Just put the name of stream there like "rds-temp-leads-stream"
Based on the error:
1 validation error detected: Value 'arn:aws:kinesis:us-west-2:xxxxxxxxxx:stream/rds-temp-leads-stream' at 'streamName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+
The highlighted value : arn:aws:kinesis:us-west-2:xxxxxxxxxx:stream/rds-temp-leads-stream doesn't match the regular expression [a-zA-Z0-9_.-]+ which causes the validation error.
This is because you do not need to add the full ARN and just add the name of your stream. i.e rds-temp-leads-stream
I'm not seeing it on the code but likely this line got the error:
kinesis.put_record(StreamName="jhgjh", Data=json.dumps(event, cls=DateTimeEncoder), PartitionKey="default")
Where "StreamName" should follow the same pattern as above. Reference can be found below: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With