Senior Data Engineer at 10pearls.

Utilizing

• Apache Spark, MapReduce, Livy, Airflow, pySpark, s3 Bucket, Snowflake, Docker.

Responsibilities

• Moving complex Structural data(SQL, mySQL, oracle, sftp) using pyspark remove redundancies and save to s3 bucket and snowflake.
• Daily jobs are monitored/scheduled using Airflow Dags.
• Spark jobs are submitted using Livy.

https://www.linkedin.com/in/vashdev-heerani-8ab205a2/