Reports To: Chief Data Architect
Location: Boston, MA
Comniscient is hiring a Data Engineer to join our Data Operations team. Utilizing proprietary big data methodologies, Comniscient provides daily carrier performance data across all markets and competitors. Our data engineers are responsible for monitoring and supporting the core ETL processes, identifying and addressing data quality issues, and representing back end operations in the new product development process.
The Data Engineer will monitor the core ETL processes using existing and new tests/processes that she or he will develop, primarily in SQL. The Data Engineer will identify data anomalies throughout the ETL process as well as potential anomalies identified by downstream users. She or he will work with the Chief Data Architect to make data and/or coding adjustments as necessary to support both ongoing operations and new product development. This person will also bring knowledge of best practices to existing processes and tools and will assist with ongoing optimization of ETL data processes and tools. This is a full-time position with significant opportunity for advancement.
Including but not limited to:
- Support the creation and analysis of Comniscient’s proprietary data
- Oversee current and new data integrity tests throughout the ETL cycle
- Identify, investigate and resolve data integrity issues
- Support and tune aggregation processes and incorporation of additional data within the ETL process and downstream
- Understand and anticipate internal and external client needs to identify potential approaches to engineering new data tools
- Collaborate with team members on customer requests and ad hoc projects
- Advanced knowledge of SQL Server/Transact-SQL and practical coding experience
- Experience working with very large data sets (i.e., multi-Tb databases)
- Experience working with complicated and/or messy data from numerous sources simultaneously
- Meticulous attention to detail and ability to document work (both within code and separately)
- Ability to work independently and to ask for help/guidance when necessary
- Drive to work enthusiastically on the sixth solution to a problem after the first five solutions fail
- Comfort working in a fast paced, collegial environment
- Tableau, R, and/or Python
- EMR packages (e.g., Hadoop, Pig, Hive) and techniques
- Familiarity with Amazon Web Services product suite
- Knowledge of (or an interest in learning) the Telecommunications industry
At this time, Comniscient will not sponsor a new applicant for employment sponsorship for this position.