IDSC4111

Modern organizations increasingly base their decisions on data which is becoming more abundant each day. The first step of using data for decision making is to prepare data in a suitable format for analysis, a step commonly known as data engineering. Typical data engineering tasks may include data acquisition, parsing, handling missing data, summarization, augmenting, transformation, subsetting, sampling, aggregation, and merging. Data engineers also frequently use basic data visualization tools to detect and fix data issues. Most recently, there is increasing demand for data engineers to handle big data and unstructured data. A good data engineering process ensures quality, reliability, and usability of data. In fact, data engineering is such a critical and time-consuming step of data-driven decision making that many data scientists and analysts spend more than 60% of their time doing data engineering related tasks.