MatchPoint Solutions is for candidates looking for a challenging career in a progressive company. MatchPoint and its clients are always looking for the qualified candidates, hiring directly out of top universities as well as experienced individuals from the industry.
Python/Sr Data Engineer_Architect
Location: San Francisco, CA
Duration: 6+ months
The Senior Data Engineer will work with the Regulatory Affairs department to build data pipelines to create a Cost of Service data lake. The resource must have programming experience with Python and other ETL tools. The resource must have experience modeling and designing a data lake. The resource must be comfortable interacting and communicating with both Business and IT project team members and needs to be a self-starter. The Data Engineer must have expertise in handling/moving large scale data sets (in tens of TB) and on to the Big data Hadoop clusters on AWS.
Computer Programming: excellent programming skill for building data pipelines from multiple data sources (relational databases, flat files, CSV, etc.). Must have good data modeling and design skills.
Must have good data modeling and design skills. – ACTUALLY building data models
Must have experience designing and modelling very high volume data.
Programming languages: Python (Primary), Must have Python experience on Hadoop, SQOOP, and other Apache tools. ETL experience must have. AWS knowledge needed.
Must have a background in Math (Calculus, Algebra)
DB: SQL (Netezza, SQL Server, Teradata), Excel
Utility or regulated industry experience (preferred)
Data analysis: excellent analytical and problem solving skills. Have proactive and strategic thinking and ability to align research with the business and innovate research output.
MUST HAVE - Python, AWS, Spark, PySpark
MUST HAVE Math background – calculous and algebra
“Client” reads meters every hour, every day in the whole year… 8,060 readings per customer and 4mil customers. 32 billion records. This person needs to do 3-7 different types of analysis (i.e., How is this being used, what time are using the most, usage records, etc.). We are looking for someone who can help design the database so it’s easier to ingest data into AWS. Break down into multiple steps, understand business problem, apply math to design database. (Not a statistics background – not trend analysis). Models are already built but need to understand what models do to design base accordingly.
Once they design data, using Python and PySpark to ingest data into AWS.
The goal is to ingest data easier and faster through the pipe.
NOT a regular developer… more of an architect level person.
Modelling data for high volume data transfer
Extract and Load experience
Data Engineers would be the correct title – someone who has been designing data, ingesting data. NOT an analyst but an Engineer/Architect.
Hadoop Developers are also Data Engineers so candidates could have that job title as well.
Technical Resource Manager | MatchPoint Solutions | Office 925-829-7755 | Cell 408-718-6170| Email email@example.com