Introduction

# Introduction

---

layout: true
  
<div class="dk-footer">
<span>
<a href="https://rfortherestofus.com/" target="_blank">R for the Rest of Us
</a>
</span>
</div>

---

class: center, middle, dk-section-title
background-image:url("images/gawn-australia-_OUvt8kLf0s-unsplash.jpg")
background-size: cover
# What is Data Cleaning?

???

Hello and welcome to Data Cleaning with R.
Before we begin, let's define Data Cleaning. Let's look at some common definitions.

---

>.large["Mechanistic, time consuming process of identifying and fixing incomplete, inaccurate, irrelevant, or duplicated data"]

???

One popular definition describes data cleaning as a mechanistic, time consuming process of identifying and fixing incomplete, inaccurate, irrelevant, or duplicated data

>.large["Tedious tasks that get in the way of extracting true value and insights from data"]

???

Others see it as an obstacle to extracting true insights from data

> .large["Not difficult, not _real_ work. Done (grudgingly) before the main analyses"]

???

and some say it is not difficult and not real work. that it is done grudgingly before the actual analyses.

---

>.large["~~.gray[Mechanistic, time consuming]~~ process of identifying and _fixing_ incomplete, inaccurate, irrelevant, or duplicated data"]

>.large["~~.gray[Tedious tasks that get in the way of]~~ extracting true value and insights from data"]

> .large["~~.gray[Not difficult, not _real_ work. Done (grudgingly) before the main analyses"]~~]

???

I'd like to refute some of that. 
yes, data cleaning means fixing issues with our data, but it is real work, and it is not trivial.

---

## Data cleaning:

.large[
A spectrum of **reusable data transformations**  to make data **usable** and to obtain more meaningful results from our analysis or visualization methods.]

???

Let's try again. We can define data cleaning as a spectrum of data transformation steps to make data usable and to get more meaningful results from our workflows.

.large[
Has no single end product, because different systems and methods consume different input data formats and structures.]

???

There is no one particular end product when we're cleaning data, because different programs take different inputs, and we need to adjust our approach for each one. Still, clean data has certan things in common that we will be discussing throughout this course.  
---

## Cleaning data involves:

???

When we need to clean our data, these are the three main steps in the process.

.large[
**1\.** Identifying specific **issues** that hinder an analysis, visualization, or reporting workflow.]

???

First, we identify the issues that hinder our workflows

???

Then we determine which approach or tool we will need

.large[
**3\.** Transforming the data, making them more appropriate and valuable for a variety of downstream purposes.
]

???

Finally, we transform the data, making them more useful for various different purposes.

---

## Common Issues

.fl.w-50[

- Non-rectangular data  
- Unusable variable names
- Whitespace 
- Inconsistent letter case  
- Missing/implicit/misplaced grouping variables
]

.fl.w-50[
- Compound values  
- Duplicated values  
- Broken values  
- Empty rows and columns  
- Numbers stored as text  
]

???

While I was preparing this class, I noticed that 
most definitions of data cleaning mention fixing issues with the data, but these issues are often mentioned without descriptions or context. I collected the most common ones here.

Data in the real world usually have these issues. This list here is for reference,but we will deal with all of them at a later section of this course.

---

## Why data cleaning

???

We often need to clean our data simply to get our software to import the data, and behave as we expect it to.

.large[
- Enables downstream tasks by making data readable & usable  
]
???
Clean data makes downstream tasks possible.

???

Having clean data saves time, for example, by having tables with rows and columns on which we can immediately perform operations.

???

After that, clean data is important to avoid delays, biases, or misinterpretations.

---

## Why data cleaning in R

???

Why should we use R for data cleaning.

Because, by working with code and not interactively with pointing and clicking, we leave a record of our actions which we can revisit whenever we want.

.large[
- Less time editing manually  
]
???
with R, we can work on large files or with many files at once, which saves us time and might not even be feasible if done by hand.

???

I also like R so much for data cleaning because there are a number of tools and packages built specifically for this, which have good documentation and a large community of users ready to help us if we get stuck.

???

Finally, the code we write can be run many times and repurposed for new projects.

---

# Examples

???

Let me show you some examples of what can happen when we don't clean our data.

---

???

This graph is fictional, but it is based on a real used by a pharmaceutical company. It shows drug sales in a year. If, for example, an executive wants to know which was the top selling drug that year, this plot can be misleading.

Why is that?

Look at the labels on the x-axis. Each drug should only appear once, but some appear twice. This is because there are small variations in how the drug names were stored. The plotting software does not realize that underscores should be the same as spaces, or that a name in upper or lower case refers to the same drug.

In this situation, the top-selling drug is not the one the chart is showing, so this is misleading. This looks obvious, but these kinds of issues are common and they can go unnoticed if we are working, for example with many years worth of data for hundreds of drugs.

---

#rrsiukcnra .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 28px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#rrsiukcnra .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#rrsiukcnra .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#rrsiukcnra .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#rrsiukcnra .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#rrsiukcnra .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#rrsiukcnra .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#rrsiukcnra .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#rrsiukcnra .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#rrsiukcnra .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#rrsiukcnra .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#rrsiukcnra .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#rrsiukcnra .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#rrsiukcnra .gt_from_md > :first-child {
  margin-top: 0;
}

#rrsiukcnra .gt_from_md > :last-child {
  margin-bottom: 0;
}

#rrsiukcnra .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#rrsiukcnra .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#rrsiukcnra .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#rrsiukcnra .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#rrsiukcnra .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#rrsiukcnra .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#rrsiukcnra .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#rrsiukcnra .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#rrsiukcnra .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#rrsiukcnra .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#rrsiukcnra .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#rrsiukcnra .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#rrsiukcnra .gt_left {
  text-align: left;
}

#rrsiukcnra .gt_center {
  text-align: center;
}

#rrsiukcnra .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#rrsiukcnra .gt_font_normal {
  font-weight: normal;
}

#rrsiukcnra .gt_font_bold {
  font-weight: bold;
}

#rrsiukcnra .gt_font_italic {
  font-style: italic;
}

#rrsiukcnra .gt_super {
  font-size: 65%;
}

#rrsiukcnra .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="rrsiukcnra" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  <thead class="gt_header">
    <tr>
      <th colspan="2" class="gt_heading gt_title gt_font_normal" style>Faculty Productivity Bonuses</th>
    </tr>
    <tr>
      <th colspan="2" class="gt_heading gt_subtitle gt_font_normal gt_bottom_border" style>2020 - School of Engineering</th>
    </tr>
  </thead>
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">Researcher</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">Publications<sup class="gt_footnote_marks">1</sup></th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left">James Wilson</td>
<td class="gt_row gt_left">19 $$$</td></tr>
    <tr><td class="gt_row gt_left">Colt Whitman</td>
<td class="gt_row gt_left">18*</td></tr>
    <tr><td class="gt_row gt_left">Jane Allen</td>
<td class="gt_row gt_left">17*</td></tr>
    <tr><td class="gt_row gt_left">Maria Dias</td>
<td class="gt_row gt_left">14</td></tr>
    <tr><td class="gt_row gt_left">Dr. Deavan Smith</td>
<td class="gt_row gt_left">9</td></tr>
    <tr><td class="gt_row gt_left">Angie Ramirez</td>
<td class="gt_row gt_left">7</td></tr>
    <tr><td class="gt_row gt_left">Ellen Dice</td>
<td class="gt_row gt_left">5</td></tr>
    <tr><td class="gt_row gt_left">Dr. Jane Allen</td>
<td class="gt_row gt_left">4</td></tr>
  </tbody>
  
  <tfoot>
    <tr class="gt_footnotes">
      <td colspan="2">
        <p class="gt_footnote">
          <sup class="gt_footnote_marks">
            <em>1</em>
          </sup>
           
          <strong>$$$</strong>  Award winner, <strong>*</strong>  Honorable mention
          <br />
        </p>
      </td>
    </tr>
  </tfoot>
</table></div>

???

This other example is also simplified and fictional, but it is based on a report used at a real university. Each year, the most productive faculty member receives a cash bonus, based on a report that is generated automatically. This is a good example of the consequences of not cleaning data before making calculations.

Here, one faculty members appears twice in the data, because for whatever reason some entries include academic titles while the rest of the entries do not. This is also a simple fix, which could have avoided some embarrassment.

---

.small[Source: [Kenya Ministry of Agriculture, Livestock and Fisheries](https://africaopendata.org/organization/kenya-ministry-of-agriculture-livestock-and-fisheries)]

???

This next example is real, describing fisheries data for a lake in Kenya in 2016. There are no mistakes in the values, but the data are structured in a way that makes sense to humans but is harder to feed into any analysis software.

If we had to restructure multiple files with this complicated layout, say, for many lakes and many years, the process can become time consuming. Fortunately, we can do this more efficiently in R, and will do so in a later lesson.

---

</br>
.left[
<img src="images/tymp.png" width="100%" />
]

???

This next example is a table from a scientific publication. A lot of information is being presented here, and let's assume the only version of these data is in a PDF file. If wanted to work with the contents of this table, we would most likely run into some trouble. 
Nothing is inherently wrong with this table, but sometimes we might be forced to import and work with these kinds of data sources

---

#cymnhrtiuq .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#cymnhrtiuq .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#cymnhrtiuq .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#cymnhrtiuq .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#cymnhrtiuq .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#cymnhrtiuq .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#cymnhrtiuq .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#cymnhrtiuq .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#cymnhrtiuq .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#cymnhrtiuq .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#cymnhrtiuq .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#cymnhrtiuq .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#cymnhrtiuq .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#cymnhrtiuq .gt_from_md > :first-child {
  margin-top: 0;
}

#cymnhrtiuq .gt_from_md > :last-child {
  margin-bottom: 0;
}

#cymnhrtiuq .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#cymnhrtiuq .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#cymnhrtiuq .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#cymnhrtiuq .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#cymnhrtiuq .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#cymnhrtiuq .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#cymnhrtiuq .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#cymnhrtiuq .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#cymnhrtiuq .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#cymnhrtiuq .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#cymnhrtiuq .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#cymnhrtiuq .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#cymnhrtiuq .gt_left {
  text-align: left;
}

#cymnhrtiuq .gt_center {
  text-align: center;
}

#cymnhrtiuq .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#cymnhrtiuq .gt_font_normal {
  font-weight: normal;
}

#cymnhrtiuq .gt_font_bold {
  font-weight: bold;
}

#cymnhrtiuq .gt_font_italic {
  font-style: italic;
}

#cymnhrtiuq .gt_super {
  font-size: 65%;
}

#cymnhrtiuq .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="cymnhrtiuq" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 18px;">mfr</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 18px;">model</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 18px;">year</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 18px;">bdy_style</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 18px;">hp</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 18px;">trq</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr class="gt_group_heading_row">
      <td colspan="6" class="gt_group_heading">Germany</td>
    </tr>
    <tr><td class="gt_row gt_left" style="font-size: 18px;">Audi</td>
<td class="gt_row gt_left" style="font-size: 18px;">R8</td>
<td class="gt_row gt_right" style="font-size: 18px;">2015</td>
<td class="gt_row gt_left" style="font-size: 18px;">coupe</td>
<td class="gt_row gt_right" style="font-size: 18px;">430</td>
<td class="gt_row gt_right" style="font-size: 18px;">317</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 18px;">Porsche</td>
<td class="gt_row gt_left" style="font-size: 18px;">911</td>
<td class="gt_row gt_right" style="font-size: 18px;">2016</td>
<td class="gt_row gt_left" style="font-size: 18px;">coupe</td>
<td class="gt_row gt_right" style="font-size: 18px;">350</td>
<td class="gt_row gt_right" style="font-size: 18px;">287</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 18px;">Mercedes-Benz</td>
<td class="gt_row gt_left" style="font-size: 18px;">AMG GT</td>
<td class="gt_row gt_right" style="font-size: 18px;">2016</td>
<td class="gt_row gt_left" style="font-size: 18px;">coupe</td>
<td class="gt_row gt_right" style="font-size: 18px;">503</td>
<td class="gt_row gt_right" style="font-size: 18px;">479</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 18px;">Audi</td>
<td class="gt_row gt_left" style="font-size: 18px;">S6</td>
<td class="gt_row gt_right" style="font-size: 18px;">2016</td>
<td class="gt_row gt_left" style="font-size: 18px;">sedan</td>
<td class="gt_row gt_right" style="font-size: 18px;">450</td>
<td class="gt_row gt_right" style="font-size: 18px;">406</td></tr>
    <tr class="gt_group_heading_row">
      <td colspan="6" class="gt_group_heading">Italy</td>
    </tr>
    <tr><td class="gt_row gt_left" style="font-size: 18px;">Ferrari</td>
<td class="gt_row gt_left" style="font-size: 18px;">458 Spider</td>
<td class="gt_row gt_right" style="font-size: 18px;">2015</td>
<td class="gt_row gt_left" style="font-size: 18px;">convertible</td>
<td class="gt_row gt_right" style="font-size: 18px;">562</td>
<td class="gt_row gt_right" style="font-size: 18px;">398</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 18px;">Ferrari</td>
<td class="gt_row gt_left" style="font-size: 18px;">LaFerrari</td>
<td class="gt_row gt_right" style="font-size: 18px;">2015</td>
<td class="gt_row gt_left" style="font-size: 18px;">coupe</td>
<td class="gt_row gt_right" style="font-size: 18px;">949</td>
<td class="gt_row gt_right" style="font-size: 18px;">664</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 18px;">Maserati</td>
<td class="gt_row gt_left" style="font-size: 18px;">Ghibli</td>
<td class="gt_row gt_right" style="font-size: 18px;">2016</td>
<td class="gt_row gt_left" style="font-size: 18px;">sedan</td>
<td class="gt_row gt_right" style="font-size: 18px;">345</td>
<td class="gt_row gt_right" style="font-size: 18px;">369</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 18px;">Maserati</td>
<td class="gt_row gt_left" style="font-size: 18px;">Quattroporte</td>
<td class="gt_row gt_right" style="font-size: 18px;">2016</td>
<td class="gt_row gt_left" style="font-size: 18px;">sedan</td>
<td class="gt_row gt_right" style="font-size: 18px;">404</td>
<td class="gt_row gt_right" style="font-size: 18px;">406</td></tr>
  </tbody>
  
  
</table></div>

???

This last example describes some fancy cars. Things are OK for the most part, but notice these rows that group the cars by the country of their maker. This is informative and readable, but not ideal from an analysis perspective, and we will see why later on in the course.

With these few examples, you hopefully were able to recognize some common issues and relate them to things you may have seen in your own work.

---

## Your turn

1. Read this short blog [post](https://www.johndcook.com/blog/2021/01/12/data-pathology/) by John D. Cook.
.center[https://www.johndcook.com/blog/2021/01/12/data-pathology/]

2. Reflect on the amount (if any) of data cleaning you perform for your day-to-day work.

???

Before we get to any code, please complete these short tasks. 
First, read a short text on why we should not consider data cleaning an obstacle to our work.
Then, reflect on how much time you spent cleaning data in your own work, do you enjoy it? do you find it tedious?

---

class: center, middle, dk-section-title
background-image:url("images/willian-justen-de-vasconcellos-JJEQ1SQhkEg-unsplash.jpg")
background-size: cover
# Data Cleaning is Analysis

???

Before moving on, I want to really emphasize that data cleaning is the first step in an analysis. Let me elaborate.

---

### When we clean data, we make judgments and interpretations

???

When we clean data, we make judgments and interpretations

- Are the data in a usable structure?

???

Can we use the data in their current state?

- What constitutes 'unwanted' variation?

???

Also, what constitutes unwanted variation?
As we saw earlier in an example, are spaces between words interchangeable with underscores? do we want to keep this kind of variation? It will always depend on the context.

- Is there any implicit information that needs to made explicit?

???

Sometimes we need to figure out if there is any information that is only implied but not actually recorded within a dataset.

Consider the example from earlier with the cars and the country of their makers. The country names show up in the data but it was up to us to interpret what this means.

- What to do about missing data.

???

Finally, what to do about missing data.  We will go over this in detail later, but it is up to us to interpret what missing values mean before moving on with a workflow.

---

## This is also data analysis.

>.large[Data cleaning is an important stage in most data analyses, not a one-off task of lesser importance.]

???

Most of the time, diagnosing issues with our data and doing something about them is the first step in data analysis.

---

.pull-left[
<blockquote class="twitter-tweet" data-dnt="true"><p lang="en" dir="ltr">I will let the data speak for itself when it cleans itself</p>&mdash; Allison Reichel (@AllisonReichel) <a href="https://twitter.com/AllisonReichel/status/1350107719236247558?ref_src=twsrc%5Etfw">January 15, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> 
]

.pull-right[
<blockquote class="twitter-tweet" data-dnt="true"><p lang="en" dir="ltr">I was recently asked, “are you sure teaching Python &amp; R is right? Isn’t the future all AI and ML?”<br><br>Munging data is going to be step 1 of analysis for the foreseeable future. <a href="https://t.co/n2tKDwI0wn">https://t.co/n2tKDwI0wn</a></p>&mdash; JD Long (@CMastication) <a href="https://twitter.com/CMastication/status/1361050957962952705?ref_src=twsrc%5Etfw">February 14, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> 
]

???

these two tweets summarize the importance of cleaning data before doing pretty much anything else. Being able to clean data is just as valuable as learning the most novel and complicated analytical methods.

---

class: center, middle, dk-section-title
background-image:url("images/endri-killo-tvWqbeOaJnw-unsplash.jpg")
background-size: cover
# The Data Cleaning Process

???

Now that we defined data cleaning, and that we recognize its importance in most workflows, lets go over the whole process.

---

???

We start the inputs. R can import pretty much every kind of file from our computer or from a cloud service. We can import the files into R and keep the raw data intact, there is no need to modify it in place.

With our data loaded into R, we can explore, restructure, and fix any issues before going on to models and plots, or to export a clean version of the data.

All the while, by working with code and scripts, we are creating a record of what we did at each step.

---
class: center, middle, dk-section-title
background-image:url("images/bence-boros-l3_9j916sh0-unsplash.jpg")
background-size: cover
# Course Logistics

???

We are almost ready to start cleaning data but before we do, lets spell out the general logistics for this course.

---

## Requirements and Prerequisites
### You:

- are familiar with basic data types and objects in R

- can use functions and arguments

- have R and RStudio running

- can install and load packages

???

Before you move on, you might want to check if you meet these basic prerequisites, like having R and Rstudio installed and ready. If you don't feel comfortable working with data types, objects, and functions, I suggest the free Fundamentals of R course.

---

## Course conventions

???
Now, some conventions that I use throughout this course.

- All packages available on **CRAN** (can be installed with .orange[`install.packages()`])

???

All the packages we will be using are available on CRAN, which should make for easier installation.

- All slides are available in html format

???

All the slides are available in html format, which you can open with any web browser.

- Package names in .b.rrured[red], preceded by 📦

???

All package names appear in red with a package emoji

- Functions and important operators in .b.orange[orange] or .b.purple[purple]

???

Important functions and operators appear in orange or purple.

- Relevant example data is provided in slides (see '**data-setup**' panels)

???

Finally, there is code for creating the example datasets that appear in the slides whenever a tabbed panel says data-setup.

---

# My setup

- R v4.0.4

- RStudio v1.4.1106

- Default theme and pane layout

- Windows 10

???

I'm running recent versions of R and Rstudio in Windows 10, and other than making the font slighly larger I made no additional customization.

---

## Course setup

**1\.** Create a new RStudio project on your computer

**2\.** Download course materials (slides and data needed for "your turn" exercises) with 
📦 .b.rrured[`usethis`]

```r
install.packages("usethis") # Install first if necessary
usethis::use_course("rfortherestofus/data-cleaning-course") # Download course materials 
```

**3\.** Copy the downloaded materials to the folder containing the RStudio project you created for this course.

???

For this course, we'll need to download some files.
First, create a new RStudio project somewhere on your computer where you will be able to find it.
Then, install the usethis package and call the use course function to download the course materials.
Finally, copy contents of the new folder into the location of the Rstudio project you created for this course.

I'll demonstrate these steps now.