SmartDataMark logo

Hey, I'm Mark
Available to help you.

I'm a seasoned Data Engineer proficient in SQL and database management, with expertise in AWS and ETL/ELT solutions using Glue. Skilled in orchestrating data workflows using Airflow, I excel in optimizing performance and automating processes. With a keen eye for detail and a passion for innovation, I'm committed to delivering robust data solutions that drive business success.

Professional Experience

  1. Data Engineer - SnapFinance

    During my tenure as a Data Engineer at Snap Finance, I have played a pivotal role in driving team success by developing and implementing innovative solutions that have enhanced the efficiency and quality of data processes. Some key benefits stemming from my work include:

  2. * Infrastructure Optimization: I developed and implemented infrastructures using technologies such as ECR and Terraform.
  3. * Process Monitoring Improvement: By enhancing Lambda functions for ETL process monitoring.
  4. * Standardization and Automation of Testing: I implemented a Quality Assurance (QA) framework allowing for standardized unit and automation testing, enhancing code reliability and consistency.
  5. * Sensitive Data Protection: I defined and applied techniques for identifying and masking Personally Identifiable Information (PII).
  6. * Data Processing Efficiency: Through the development of Glue jobs with PySpark, I optimized data cleaning, transformation, and loading into parquet formats in S3 buckets, improving efficiency and reducing processing times.
  7. * Transition to Microservices Architecture: Utilizing technologies like Kinesis, I facilitated the transition from monolithic applications to microservices architectures, enabling greater flexibility, scalability, and maintainability of the system.

  8. Data Engineer - Accenture

    I played a pivotal role in supporting data workflows and optimizing data processes to meet project requirements. Some key responsibilities and contributions include

  9. * Workflow Monitoring: I was responsible for monitoring data workflows using Informatica, ensuring the smooth execution of data pipelines and identifying and resolving any issues or bottlenecks.
  10. * HQL File Management: I took charge of fixing HQL (Hive Query Language) files, addressing any errors or inconsistencies to ensure the accuracy and efficiency of data processing tasks.

  11. DBA - OMNI

    I played a crucial role in managing and optimizing database environments to ensure efficiency, reliability, and performance.

  12. * Environment Setup: I was responsible for creating database environments using AWS RDS (Relational Database Service), ensuring the proper configuration and setup to meet business requirements.
  13. * Performance Optimization: I focused on enhancing the performance of stored procedures, implementing optimizations to improve query execution times and overall database efficiency.
  14. * Database Design: I played a key role in designing database schemas and diagrams, ensuring scalability, data integrity, and optimal performance.
  15. * Database Migration: I led database migration efforts, overseeing the seamless transition of data from legacy systems to modernized environments, minimizing downtime and ensuring data integrity throughout the process. Backup and Recovery: I implemented robust backup and restore procedures for databases hosted on AWS RDS, ensuring data protection and disaster recovery readiness.
  16. * Schema Improvement: I actively worked on improving database schemas, optimizing data structures and relationships to enhance performance and facilitate future scalability.

Services

Data Engineering Services

  • • Infrastructure setup and management using AWS services such as RDS, Glue, and Kinesis.
  • • Design and implementation of data pipelines using tools like PySpark and Apache Airflow.
  • • ETL (Extract, Transform, Load) process development and optimization.
  • • Data migration between different databases or cloud environments.
  • • Implementation of data quality assurance frameworks and automated testing using Python and PyTest/UnitTest.
  • • Real-time data processing and stream processing with technologies like Kinesis.

Database Administration Services

  • • Setup, configuration, and management of relational databases such as AWS RDS.
  • • Performance tuning and optimization of SQL queries and stored procedures.
  • • Database schema design and optimization for improved performance and scalability.
  • • Database migration planning and execution, ensuring minimal downtime and data integrity.
  • • Implementation of backup and recovery strategies to safeguard data.
  • • Identification and masking of Personally Identifiable Information (PII) to ensure compliance with data privacy regulations.

Consultation and Advisory Services

  • • Advising clients on best practices for data engineering, database management, and data architecture.
  • • Providing guidance on technology selection, architecture design, and implementation strategies.
  • • Performing assessments and audits of existing data systems to identify areas for improvement and optimization.
  • • Offering training and knowledge transfer to client teams on data engineering and database management practices.

Contact me

Got a technical issue? Need details to make Business? Let me know.