Common Issues in Data Cleaning

# Common Issues in Data Cleaning

---

layout: true
  
<div class="dk-footer">
<span>
<a href="https://rfortherestofus.com/" target="_blank">R for the Rest of Us
</a>
</span>
</div>

---

class: center, middle, dk-section-title
background-image:url("images/michael-We2t3suGiYk-unsplash.jpg")
background-size: cover
# Common Issues in Real-World Data and Their Consequences

???

In this lesson, we’ll go over most common issues that we encounter in real-world data. The files and datasets that we receive or come across for day-to-day use are never as usable as the ones we use when we’re learning.

So, in the following sections, we will learn how to identify these issues, and more importantly, how to fix them.

---

## 🌍 Real-world data

???

The thing with real-world data is that files and tables vary in how messy they are, and these issues can vary in how much they affect our workflows.

- May not be readily imported

???

In some cases we might not be able to load the data into our software of choice,

- May not function properly within analysis software

???

or maybe we can, but they will not be immediately usable and we will not be able to get things done in the first place.

- Often contains issues that go unnoticed until they:

- break a workflow
  - introduce biases
  - waste resources (computational, $)

???

Alternatively, real world data can have small issues that can go unnoticed until 
they either break something, or they introduce biases and errors. At this point it is likely that going back and fixing these issues will mean wasting time or resources.

The good news is that most issues can be fixed before they trip up any data analysis or visualization, and simply by being aware of the most common issues we can make data usable more efficiently.

---

## General data cleaning workflows

???

There is no single sequence for cleaning data, but I recommend this general workflow because these broad steps make our data usable in incremental steps.

???

The first prerequisite would be to make sure we have rectangular data, with usable rows and columns.

???

After that, it is advisable to have column names that mean something to us, and preferably with a consistent naming scheme.

???

For the rows in our data, we want to be able to identify the observational units, and have one row for each observation.

???

Once we have usable rows and columns, then we can make sure that any variable that aggregates our observations into groups is doing so adequately.

---

## Less interpretation

???

Although there is no specific recipe to follow for cleaning data, there are some steps that we can do first because they will not interfere with the rest of our work. If we recognize any of these issues in our data, we can usually fix them first because they need less interpretation from us about why these issues are present in the first place.

???

For example, extra spaces between words are rarely intentional and present in our data because they mean something. More likely, they are typing mistakes or artifacts from prior data transformations, so we can just get rid of any spaces that we determine to be unwanted. This can also apply to inconsistencies in letter case, and unless we are specifically interested in how capitalization varies within our data, we can usually fix this straight away.  Same with duplicate entries.

## More interpretation

???

On the other hand, there are issues in our data for which we need to stop and make a more careful interpretation.

--
.large[
Cleaning numeric variables, unbreaking values, extracting target data
]

???

For example, we might need to figure out what it means if a numeric variable has inconsistent decimal symbols or annotations, or if values are broken across more than one row or column, what criteria were used to break them. This will become clear as we go over the most common issues.

---
class: center, middle, dk-section-title
background-image:url("images/james-rathmell-t0iwmK0WC0Y-unsplash.jpg")
background-size: cover
# Unusable Headers

???

Whether our data has a only handful of rows and columns, or millions of observations for hundreds of columns, we need ways of referring to each one. To refer efficiently to 2 or 2000 columns, we need them to have usable, informative names.

The names for the columns in our data are in the first row, or the header. If the headers have issues, we need to address them before we proceed.  This is why sorting out headers is the first general step in our workflow.

---

## Unusable headers

???

There are many ways for headers to be unusable.

Inconsistent or uninformative names

#tcfldvpgyk .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#tcfldvpgyk .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#tcfldvpgyk .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#tcfldvpgyk .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#tcfldvpgyk .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#tcfldvpgyk .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#tcfldvpgyk .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#tcfldvpgyk .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#tcfldvpgyk .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#tcfldvpgyk .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#tcfldvpgyk .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#tcfldvpgyk .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#tcfldvpgyk .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#tcfldvpgyk .gt_from_md > :first-child {
  margin-top: 0;
}

#tcfldvpgyk .gt_from_md > :last-child {
  margin-bottom: 0;
}

#tcfldvpgyk .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#tcfldvpgyk .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#tcfldvpgyk .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#tcfldvpgyk .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#tcfldvpgyk .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#tcfldvpgyk .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#tcfldvpgyk .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#tcfldvpgyk .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#tcfldvpgyk .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#tcfldvpgyk .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#tcfldvpgyk .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#tcfldvpgyk .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#tcfldvpgyk .gt_left {
  text-align: left;
}

#tcfldvpgyk .gt_center {
  text-align: center;
}

#tcfldvpgyk .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#tcfldvpgyk .gt_font_normal {
  font-weight: normal;
}

#tcfldvpgyk .gt_font_bold {
  font-weight: bold;
}

#tcfldvpgyk .gt_font_italic {
  font-style: italic;
}

#tcfldvpgyk .gt_super {
  font-size: 65%;
}

#tcfldvpgyk .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="tcfldvpgyk" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 17px;">X</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 17px;">X1</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 17px;">X2</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 17px;">mean_Score</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 17px;">AVERAGE.SCORE</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left" style="font-size: 17px;">UMN</td>
<td class="gt_row gt_left" style="font-size: 17px;">EAST</td>
<td class="gt_row gt_left" style="font-size: 17px;">A</td>
<td class="gt_row gt_right" style="font-size: 17px;">7.7</td>
<td class="gt_row gt_right" style="font-size: 17px;">7.701</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 17px;">UV</td>
<td class="gt_row gt_left" style="font-size: 17px;">WEST</td>
<td class="gt_row gt_left" style="font-size: 17px;">B</td>
<td class="gt_row gt_right" style="font-size: 17px;">8.9</td>
<td class="gt_row gt_right" style="font-size: 17px;">8.890</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 17px;">UNLV</td>
<td class="gt_row gt_left" style="font-size: 17px;">EAST</td>
<td class="gt_row gt_left" style="font-size: 17px;">C</td>
<td class="gt_row gt_right" style="font-size: 17px;">9.2</td>
<td class="gt_row gt_right" style="font-size: 17px;">9.199</td></tr>
  </tbody>
  
  
</table></div>

???

The most common issue with headers is when they have Inconsistent or uninformative names.

- Distinct != informative

???

In this example, names in the header are all distinct, but they do not tell us much. One name has a dot between words and one has an underscore.

- More difficult to remember and specify

???

Names like these are more difficult to remember, and to refer to them in functions.

- Do not sort well

???

Another consequence of inconsistent naming is that columns will not sort well if we want to sort them alphabetically.

Inconsistent names are fixable with string manipulation, which we will do later on.

---

## Unusable headers

???
Another common issue that makes headers unusable is when column names are broken across rows.

#ggvmcjgkfo .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#ggvmcjgkfo .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#ggvmcjgkfo .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#ggvmcjgkfo .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#ggvmcjgkfo .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#ggvmcjgkfo .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#ggvmcjgkfo .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#ggvmcjgkfo .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#ggvmcjgkfo .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#ggvmcjgkfo .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#ggvmcjgkfo .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#ggvmcjgkfo .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#ggvmcjgkfo .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#ggvmcjgkfo .gt_from_md > :first-child {
  margin-top: 0;
}

#ggvmcjgkfo .gt_from_md > :last-child {
  margin-bottom: 0;
}

#ggvmcjgkfo .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#ggvmcjgkfo .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#ggvmcjgkfo .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#ggvmcjgkfo .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#ggvmcjgkfo .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#ggvmcjgkfo .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#ggvmcjgkfo .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#ggvmcjgkfo .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#ggvmcjgkfo .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#ggvmcjgkfo .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#ggvmcjgkfo .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#ggvmcjgkfo .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#ggvmcjgkfo .gt_left {
  text-align: left;
}

#ggvmcjgkfo .gt_center {
  text-align: center;
}

#ggvmcjgkfo .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#ggvmcjgkfo .gt_font_normal {
  font-weight: normal;
}

#ggvmcjgkfo .gt_font_bold {
  font-weight: bold;
}

#ggvmcjgkfo .gt_font_italic {
  font-style: italic;
}

#ggvmcjgkfo .gt_super {
  font-size: 65%;
}

#ggvmcjgkfo .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="ggvmcjgkfo" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 17px;">X</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 17px;">X1</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 17px;">X2</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 17px;">mean</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 17px;">AVERAGE</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left" style="font-size: 17px;">NA</td>
<td class="gt_row gt_left" style="font-size: 17px;">NA</td>
<td class="gt_row gt_left" style="font-size: 17px;">NA</td>
<td class="gt_row gt_left" style="font-size: 17px;">Score</td>
<td class="gt_row gt_left" style="font-size: 17px;">SCORE</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 17px;">UMN</td>
<td class="gt_row gt_left" style="font-size: 17px;">EAST</td>
<td class="gt_row gt_left" style="font-size: 17px;">A</td>
<td class="gt_row gt_left" style="font-size: 17px;">7.7</td>
<td class="gt_row gt_left" style="font-size: 17px;">7.701</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 17px;">UV</td>
<td class="gt_row gt_left" style="font-size: 17px;">WEST</td>
<td class="gt_row gt_left" style="font-size: 17px;">B</td>
<td class="gt_row gt_left" style="font-size: 17px;">8.9</td>
<td class="gt_row gt_left" style="font-size: 17px;">8.89</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 17px;">UNLV</td>
<td class="gt_row gt_left" style="font-size: 17px;">EAST</td>
<td class="gt_row gt_left" style="font-size: 17px;">C</td>
<td class="gt_row gt_left" style="font-size: 17px;">9.2</td>
<td class="gt_row gt_left" style="font-size: 17px;">9.199</td></tr>
  </tbody>
  
  
</table></div>
]

???

This means that the header row only contains part of the column names, and the remaining parts are embedded in the first few rows, which should only contain data. I’ve seen this a lot when working with spreadsheets.

???

As we can see in this example, variable names appear outside of the header row

???

and fragments of the headers are mixed in with data rows.

???

The separator between fragments is nowhere to be seen, so we dont know if it was originally a space, a dot, a dash, underscore, or nothing.

???

To make things worse, there are NA or empty values introduced for names that aren’t broken.

This can also fixed, but before we do so, let’s go over some good practices for naming variables.

---

## Cleaning column names

???

In the data organization section, we discussed good names for files, objects, and variables, but lets go a little deeper

### Syntactically valid variable names

???

Syntactically valid means that R will understand these names, and that they will not cause any functions to fail.

- Contain only letters, numbers, dots or underscores

???

valid names should not have special characters,

- Start with a letter or a dot (not followed by a number)

???

and they should start with a letter or a dot.

- Not reserved words (_if_, _else_, _for_, _in_, _TRUE_, _NaN_, etc.)

???

Also, some words are reserved for other programming purposes so they are not allowed in names.

Many of the functions we use to create tables, data frames, and variables protect us from making invalid names, but we still need them to be informative and consistent.

---

## Cleaning column names

???

Knowing how to name things is important, but it is also important to learn how clean names in existing data

???

To clean existing column names, we can perform string manipulations and use regular expressions to clean and replace the names,

- Strips special characters

- Changes spaces and dots to underscores

- User-defined capitalization (default is snake_case)

]

???

or we can use an existing user friendly tool

the JANITOR package has a useful purpose-built function for cleaning names.

The clean names function will strip any special characters, replace spaces and dots with underscores, and apply a consistent letter case.

---

```r
badnames
```

```
## # A tibble: 3 x 5
##   X     X1    X2    mean_Score AVERAGE.SCORE
##   <chr> <chr> <chr>      <dbl>         <dbl>
## 1 UMN   EAST  A            7.7          7.70
## 2 UV    WEST  B            8.9          8.89
## 3 UNLV  EAST  C            9.2          9.20
```

???

The names in this example dataset are kind of a mess, with inconsistent character case and separators beween words.

```r
badnames %>%
  clean_names()
```

```
## # A tibble: 3 x 5
##   x     x1    x2    mean_score average_score
##   <chr> <chr> <chr>      <dbl>         <dbl>
## 1 UMN   EAST  A            7.7          7.70
## 2 UV    WEST  B            8.9          8.89
## 3 UNLV  EAST  C            9.2          9.20
```

???

To fix this, the tibble object called badnames is piped into the clean names function, with no additional arguments. In the resulting object all the names have consistent spacing and letter case.

---

## Broken headers

???
Clean names make headers usable, but if they are broken across rows, we can also use a purpose-built tool to put them together.

```r
badheaders
```

```
## # A tibble: 4 x 5
##   X     X1    X2    mean  AVERAGE
##   <chr> <chr> <chr> <chr> <chr>  
## 1 <NA>  <NA>  <NA>  Score SCORE  
## 2 UMN   EAST  A     7.7   7.701  
## 3 UV    WEST  B     8.9   8.89   
## 4 UNLV  EAST  C     9.2   9.199
```

???

This tibble object called badheaders has its headers broken across rows, specifically the last two columns in which mean score and average score are separated across rows, with NAs introduced in the first three columns. Let's see how we can fix this.

---

## Broken headers

.large[
Mash the top _n_ data rows column-wise with .b.purple[`mash_colnames()`] from 📦 .rrured.b[`unheadr`]
]

???

If we come across broken headers, we can use the mash colnames function from the unheadr package.

```r
badheaders %>%
  mash_colnames(
*   n_name_rows = 1, # number of data rows with header fragments
    sep = "_") # separator for collapsing header fragments
```

> Data rows exclude the header row

???

Mash colnames takes a data object, and it has an n name rows argument, to specify how many data rows have name fragments, we can also specify what to use as a separator when collapsing the header fragments into a single string.

Remember though, that data rows do not include the header row.
---

```r
badheaders %>%
  mash_colnames(n_name_rows = 1,
    sep = "_")
```

```
## # A tibble: 3 x 5
##   X     X1    X2    mean_Score AVERAGE_SCORE
##   <chr> <chr> <chr> <chr>      <chr>        
## 1 UMN   EAST  A     7.7        7.701        
## 2 UV    WEST  B     8.9        8.89         
## 3 UNLV  EAST  C     9.2        9.199
```

???

In this case there is one row with parts of the names, and we want to use an underscore for the new names.

```r
badheaders %>%
  mash_colnames(n_name_rows = 1,
    sep = "_") %>%
  clean_names()
```

```
## # A tibble: 3 x 5
##   x     x1    x2    mean_score average_score
##   <chr> <chr> <chr> <chr>      <chr>        
## 1 UMN   EAST  A     7.7        7.701        
## 2 UV    WEST  B     8.9        8.89         
## 3 UNLV  EAST  C     9.2        9.199
```

???

After that, we can pipe the resulting object into clean names, which will give us consistent letter case.

---

---

## Your turn

---
class: center, middle, dk-section-title
background-image:url("images/farnoosh-abdollahi-vIkABUsLEDY-unsplash.jpg")
background-size: cover
# Whitespace

???

Once we have data with usable names, we can deal with the seemingly minor issue of white space.
---

## Whitespace

???

Spaces are not visible characters that print on our screen, but they still take up space in a string.

This can be a problem when there are extra spaces and we cant see them, let's look at some examples of how this can affect our work.

```r
string_a <- "This string starts with a T"
string_b <- " This string starts with a space"
string_c <- "This string has trailing whitespace "
tibble(strings = c(string_a, string_b, string_c))
```

```
## # A tibble: 3 x 1
##   strings                               
##   <chr>                                 
## 1 "This string starts with a T"         
## 2 " This string starts with a space"    
## 3 "This string has trailing whitespace "
```

???

Of these three character strings, two of them have extra whitespace, and if we put them in a rectangular object, these extra spaces can be hard to see and likely to cause trouble. Let's see.

---

```r
string_a <- "R for the rest of us"
string_b <- " R for the rest of us"
string_c <- "R for the rest of us "
string_d <- " R for the rest of us "
string_e <- "R for the   rest of us"
RrU <- tibble(strings = c(string_a, string_b, string_c, string_d, string_e))
RrU
```

```
## # A tibble: 5 x 1
##   strings                 
##   <chr>                   
## 1 "R for the rest of us"  
## 2 " R for the rest of us" 
## 3 "R for the rest of us " 
## 4 " R for the rest of us "
## 5 "R for the   rest of us"
```

???

Here, these five character vectors all have say 'R for the Rest of us', but I added extra spaces at different positions.

---

```r
RrU %>%
  distinct(strings)
```

???

Because of the spaces, when we use the distinct function to keep only the unique values in the data, we get back all the rows. This is because to the computer, each string is different, even when they all say the same thing, which is R for the Rest of Us.

---

## Whitespace Issues

Extra spaces can be anywhere in a string, but we can usually describe them as being in one of these three positions:

???
whitespace can be anywhere in a sting, but we can usually find it
--

???

At the beginning of a string, or leading whitespace

???

or trailing at the end of a string

???

or alongside existing spaces between words in a string

---

## Diagnosing whitespace issues

???

Lets look at the example from earlier.
--

```
## # A tibble: 5 x 1
##   strings                 
##   <chr>                   
## 1 "R for the rest of us"  
## 2 " R for the rest of us" 
## 3 "R for the rest of us " 
## 4 " R for the rest of us "
## 5 "R for the   rest of us"
```
]

???

In the string R for the rest of us, there are six words so there should only be five spaces between these words. Remember though that the data we created earlier had extra spaces in various spots. Just looking at the tibble, the text seems misaligned, so we know that the spacing is off.

```r
str_count(RrU$strings, "\\w+")
```

```
## [1] 6 6 6 6 6
```

```r
str_count(RrU$strings, "\\s")
```

```
## [1] 5 6 6 7 7
```
]

???

This is just for demonstration, but one option for diagnosing whitespace issues could be to count the number of words and spaces in each string. For this, we can use the str count function from stringr with regular expressions. One to match words, and one to match spaces. We know there should be five spaces, so when we get the number of matches for each element and some have more than five, then we know that there is extra space that we might want to remove.

---

### Ambiguous strings

???

Another way we can tell if there are any leading or trailing spaces in a column of type character, is by printing the object to the console. This only applies if we are working with tibble objects, and we usually are because tidyverse functions typically produce tibbles.

**.purple[`is.ambiguous_string()`]**

???

If we print our objects in the console and there are leading or trailing spaces in our strings, then all the values in the columns with ambiguous strings will be quoted.

.large[
- Unexported function from 📦 .rrured.b[`pillar`]
- Used by 📦 .rrured.b[`tibble`] to reveal ambiguous strings when printing to console
]

???

This happens because in the background, the printing method for tibbles calls on the is.ambiguous.string function, from the pillar package, and if a string is ambiguous, we'll see it quoted.

```r
function(x) {
  !is.na(x) & grepl("^$|^ | $|\\\\|\"", x)
}
```

???

But what makes a string ambiguous? We can look at the function definition and realize that ambiguous strings are defined with a regular expression. We can say that a string is ambiguous if it's empty, has leading or trailing whitespace, or has unmatched quotes or escapes.

Lets try it out

---

```r
RrU %>% mutate(is_ambiguous = pillar:::is_ambiguous_string(strings))
```

```
## # A tibble: 5 x 2
##   strings                  is_ambiguous
##   <chr>                    <lgl>       
## 1 "R for the rest of us"   FALSE       
## 2 " R for the rest of us"  TRUE        
## 3 "R for the rest of us "  TRUE        
## 4 " R for the rest of us " TRUE        
## 5 "R for the   rest of us" FALSE
```

```r
RrU %>% mutate(is_ambiguous = str_detect(strings, "^$|^ | $"))
```

???

here we're using mutate to add a new column to our RRU object, which will label if the strings are ambiguous or not.

The first approach uses the is.ambiguous.string function, and in the second one we're using str detect and a regular expression for empty strings or strings with either leading or trailing whitespace.

The results are the same for both approaches, but notice that the last row has some evident extra spaces, and it is not recognized as an ambiguous string under this definition. We can go further.

---

## Removing whitespace: .b.purple[`trim_ws`]

???

We know that we can identify strings with leading and trailing spaces. We could use string manipulation to find and remove these, but fortunately there are existing tools for cleaning spaces. Our first tool is trim ws, which is short for whitespace.

???

Trim ws is not a function, but an argument for other functions in the readr package. Readr itself is a tidyverse package for importing data into R.

???

When we import files into R with functions from readr, leading and trailing whitespace can be removed automatically.

`TRUE` by default for **`read_csv()`** and **`read_tsv()`**

???

this is the default behavior when importing csv or tsv files

`FALSE` by default for **`read_delim()`**

???

but not when importing other delimited text files. This behavior in readr is already a big help during data import, and it's also a good thing that this is option. However, we still need to deal with repeated whitespace inside a a string.

---

## Removing whitespace: .b.purple[`str_squish()`]

???

For this, we can use str squish

String manipulation function from 📦 **.rrured.b[`stringr`]**

Removes whitespace from the start and end of a string and reduces repeated whitespace inside the string

???

str squish is a string manipulation function from stringr, it will remove leading and trailing spaces, and also repeated spaces inside the string.

```r
RrU %>%
  mutate(strings_sq = str_squish(strings))
```

```
## # A tibble: 5 x 2
##   strings                  strings_sq          
##   <chr>                    <chr>               
## 1 "R for the rest of us"   R for the rest of us
## 2 " R for the rest of us"  R for the rest of us
## 3 "R for the rest of us "  R for the rest of us
## 4 " R for the rest of us " R for the rest of us
## 5 "R for the   rest of us" R for the rest of us
```

???

lets try it out. Here we'll use it inside mutate from dplyr to create a new column without any extra spaces.

Notice two things in the new column. First, the strings are not quoted anymore, which is a subtle way of telling us that they are not ambiguous. And second, also see how the text just lines up.

These visual cues are good for small datasets, but not necessarily useful for larger datasets with more rows.

---

## Count words and spaces (again)

**"R for the rest of us"** (six words, five spaces)

???

After squishing the strings, we can check again if the leading and trailing spaces, as well as the repeated interior whitespace, was removed.

```r
RrU %>%
  mutate(strings_sq = str_squish(strings)) %>%
  mutate(n_words = str_count(strings_sq, "\\w+"),
    n_spaces = str_count(strings_sq, "\\s"))
```

```
## # A tibble: 5 x 4
##   strings                  strings_sq           n_words n_spaces
##   <chr>                    <chr>                  <int>    <int>
## 1 "R for the rest of us"   R for the rest of us       6        5
## 2 " R for the rest of us"  R for the rest of us       6        5
## 3 "R for the rest of us "  R for the rest of us       6        5
## 4 " R for the rest of us " R for the rest of us       6        5
## 5 "R for the   rest of us" R for the rest of us       6        5
```

???

and, we get the same numbers for all the rows.
Counting words and numbers was just for demonstration. A more realistic and useful approach would be to count the number of unique strings in our column of interest after we fix white space issues.

---

## Count distinct strings after removing extra whitespace

???

Like we did earlier, we can use the distinct function from dply to remove duplicates, and in this case we'll get back a single distinct value.

```r
RrU %>%
  mutate(strings = str_squish(strings)) %>%
  distinct()
```

```
## # A tibble: 1 x 1
##   strings             
##   <chr>               
## 1 R for the rest of us
```

???

Overall, whitespace issues can be very annoying, but their fix is relatively easy. If we use the readr package for data import, we'll trim leading and trailing spaces. At other stages of our work, if we need to remove extra spaces included repeated spaces inside strings, str squish will handle this without problem.

---

# My turn

.large[
- In the Marine Protected Areas dataset from the previous lesson (MPAS-mine.csv), check the _geographic_location_ variable for leading or trailing whitespace and remove it if necessary.
]

---

# Your turn

.large[
- In the Marine Protected Areas dataset from the previous lesson (MPAS-your.csv), check the _Country_ variable for leading or trailing whitespace and remove it if necessary.
]

---

class: center, middle, dk-section-title
background-image:url("images/snake-mylene2401.jpg")
background-size: fill
# Letter Case

???

The next common issue we'll address is letter case. Letter case issues can trip up our analyses, but fortunately fixing this is not too difficult. LEts start.

---

## Letter case

### 🧶 Strings in R are ⌨️ case sensitive

???

As you may remember from an earlier lesson, strings in R are case sensitive, and to R the same letters in upper and lower case are not interchangeable.

Addressing letter case issues:

???
There are many good reasons to spend a little bit of time dealing with letter at the early stages of our work.

- Removes unwanted variation

???
First, having consistent letter case will remove unwanted variation,

- Improves noise/signal ratio

???
This improves our signal to noise ratio,
--

- Adds consistency

???
makes the data more consistent.

- Improves readability (more shape contrast in lower vs. upper case, for many popular fonts)

???

Fixing lettercase issues can also make our data more readable, because lower and upper case letters differ in how easily we can tell them apart.

---

.panelset[
.panel[.panel-name[Districts]
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#alprcxrybm .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #f4f4f9;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#alprcxrybm .gt_heading {
  background-color: #f4f4f9;
  text-align: center;
  border-bottom-color: #f4f4f9;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#alprcxrybm .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #f4f4f9;
  border-bottom-width: 0;
}

#alprcxrybm .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #f4f4f9;
  border-top-width: 0;
}

#alprcxrybm .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#alprcxrybm .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#alprcxrybm .gt_col_heading {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#alprcxrybm .gt_column_spanner_outer {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#alprcxrybm .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#alprcxrybm .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#alprcxrybm .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#alprcxrybm .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#alprcxrybm .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#alprcxrybm .gt_from_md > :first-child {
  margin-top: 0;
}

#alprcxrybm .gt_from_md > :last-child {
  margin-bottom: 0;
}

#alprcxrybm .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#alprcxrybm .gt_stub {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#alprcxrybm .gt_summary_row {
  color: #333333;
  background-color: #f4f4f9;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#alprcxrybm .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#alprcxrybm .gt_grand_summary_row {
  color: #333333;
  background-color: #f4f4f9;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#alprcxrybm .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#alprcxrybm .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#alprcxrybm .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#alprcxrybm .gt_footnotes {
  color: #333333;
  background-color: #f4f4f9;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#alprcxrybm .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#alprcxrybm .gt_sourcenotes {
  color: #333333;
  background-color: #f4f4f9;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#alprcxrybm .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#alprcxrybm .gt_left {
  text-align: left;
}

#alprcxrybm .gt_center {
  text-align: center;
}

#alprcxrybm .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#alprcxrybm .gt_font_normal {
  font-weight: normal;
}

#alprcxrybm .gt_font_bold {
  font-weight: bold;
}

#alprcxrybm .gt_font_italic {
  font-style: italic;
}

#alprcxrybm .gt_super {
  font-size: 65%;
}

#alprcxrybm .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="alprcxrybm" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 22px;">District</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 22px;">Contribution.(USD)</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 22px;">Constituents</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left" style="font-size: 22px;">Orange walk</td>
<td class="gt_row gt_right" style="font-size: 22px;">10990</td>
<td class="gt_row gt_right" style="font-size: 22px;">12</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 22px;">TOLEDO</td>
<td class="gt_row gt_right" style="font-size: 22px;">30000</td>
<td class="gt_row gt_right" style="font-size: 22px;">7</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 22px;">Stann Creek</td>
<td class="gt_row gt_right" style="font-size: 22px;">3400</td>
<td class="gt_row gt_right" style="font-size: 22px;">9</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 22px;">Toledo</td>
<td class="gt_row gt_right" style="font-size: 22px;">21999</td>
<td class="gt_row gt_right" style="font-size: 22px;">7</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 22px;">Orange WalK</td>
<td class="gt_row gt_right" style="font-size: 22px;">8800</td>
<td class="gt_row gt_right" style="font-size: 22px;">12</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 22px;">Orange Walk</td>
<td class="gt_row gt_right" style="font-size: 22px;">800</td>
<td class="gt_row gt_right" style="font-size: 22px;">12</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 22px;">Orange Walk</td>
<td class="gt_row gt_right" style="font-size: 22px;">55000</td>
<td class="gt_row gt_right" style="font-size: 22px;">12</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 22px;">stann creek</td>
<td class="gt_row gt_right" style="font-size: 22px;">22999</td>
<td class="gt_row gt_right" style="font-size: 22px;">9</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 22px;">Toledo</td>
<td class="gt_row gt_right" style="font-size: 22px;">4900</td>
<td class="gt_row gt_right" style="font-size: 22px;">7</td></tr>
  </tbody>
  
  
</table></div>
]

```r
districts <- tibble::tribble(
  ~District, ~`Contribution.(USD)`, ~Constituents,
  "Orange walk", 10990L, 12L,
  "TOLEDO", 30000L, 7L,
  "Stann Creek", 3400L, 9L,
  "Toledo", 21999L, 7L,
  "Orange WalK", 8800L, 12L,
  "Orange Walk", 800L, 12L,
  "Orange Walk", 55000L, 12L,
  "stann creek", 22999L, 9L,
  "Toledo", 4900L, 7L
)
```
]
]

???

These will be our example data for this lesson. It's a table with three variables, describing some made up electoral data for three districts in Belize.

With these data, lets say we want to simply calculate the total contributions by district.

---

## Total contributions by district

???
In our tidyverse-oriented approach, we would group the data by district, then use the summarize function to add up the contribution variable for each group. This would go into a new variable, which we can call total underscore contribution.

```r
districts %>%
  group_by(District) %>%
  summarize(total_contribution = sum(`Contribution.(USD)`))
```

```
## # A tibble: 7 x 2
##   District    total_contribution
## * <chr>                    <int>
## 1 Orange walk              10990
## 2 Orange Walk              55800
## 3 Orange WalK               8800
## 4 stann creek              22999
## 5 Stann Creek               3400
## 6 Toledo                   26899
## 7 TOLEDO                   30000
```

???

When we run this, we get duplicates and we're not arriving at a single sum for each of the three districts. The districts variable is not quoted, and we know now that this means that there are no extra spaces. In this case, the duplication is coming from variation and inconsistency in the letter case.

---

## Detecting lower and upper case

???
The variation in letter case in these district names is more or less evident. For demonstration, we can use regular expressions to label the strings that are all uppercase and those that are all lowercase. Anything else would be mixed case.

```r
districts %>%
  mutate(
    lowercase = str_detect(District, "^[a-z\\s]+$"),
    uppercase = str_detect(District, "^[A-Z\\s]+$")
  ) %>%
  select(District, lowercase, uppercase)
```

```
## # A tibble: 9 x 3
##   District    lowercase uppercase
##   <chr>       <lgl>     <lgl>    
## 1 Orange walk FALSE     FALSE    
## 2 TOLEDO      FALSE     TRUE     
## 3 Stann Creek FALSE     FALSE    
## 4 Toledo      FALSE     FALSE    
## 5 Orange WalK FALSE     FALSE    
## 6 Orange Walk FALSE     FALSE    
## 7 Orange Walk FALSE     FALSE    
## 8 stann creek TRUE      FALSE    
## 9 Toledo      FALSE     FALSE
```

???
The regex here looks complicated but it actually is pretty simple. We're looking for character ranges for either all lowercase letters and spaces or all uppercase letters and spaces, with anchors to specify that we want this match across the whole string.

In the result, we see that there is one district name in all caps, one in all lowercase, and the rest is in mixed case. This is progress, but for clean data we actually need to clean these district names. Thankfully, there are also tools built specifically for this.

---

## Changing case with 📦 .rrured.b[`snakecase`]

???
When our data has variations in letter case, we can go ahead and change case with the snakecase package.

```r
to_any_case(strings, case)
```

???

This is all powered by the to any case function, which takes a vector of strings, and the letter case that we want to change them into. We can choose from a number of existing target cases built into the package. These include

**snake_case** - all lowercase, underscores between words

**slug-case** - all lowercase, dashes between words  
]

**Sentence case** - First word capitalized, spaces between words  
]

???

camelCase, title case, sentence case, slug case, or 
snake case, which is a very sensible default, because it uses underscores between words and all lowercase letters which are easy to read

---

## Convert case first with .b.purple[`to_any_case()`]

???
Let's summarize the total contributions again, but this time after convertin the letter case.

```r
districts %>%
  mutate(District_clean = to_any_case(District, "title",
    parsing_option = 0 
  )) %>%
  group_by(District_clean) %>%
  summarize(total_contribution = sum(`Contribution.(USD)`))
```

```
## # A tibble: 3 x 2
##   District_clean total_contribution
## * <chr>                       <int>
## 1 Orange Walk                 75590
## 2 Stann Creek                 26399
## 3 Toledo                      56899
```

> .b.purple[`?to_any_case()`] for details on parsing options

???
We're using to any case inside mutate to create a new variable called district clean, before summarizing the data. We selected title as our target case because it capitalizes all words, and these districts are proper names that should be capitalized like this.

This now gives us a clean, accurate summary of the totals.

You might have noticed an argument for the to any case function called parsing option. This is for determining how the function interprets changes in letter case to see if mixed case words need to be separated. In our case, cero means to do no parsing.

---

## 📦 .rrured.b[`snakecase`] shortcuts

???

The snakecase package comes with additional functions that do the same thing as to any case, but the target case is built into the name.

```r
districts %>%
  mutate(District_clean = to_title_case(District, 
    parsing_option = 0
  )) %>%
  group_by(District_clean) %>%
  summarize(total_contribution = sum(`Contribution.(USD)`))
```

???
In this example, we're using the to_title_case function to convert the districts into title case, without having to specify it separately.

---

.large[
- Import the Marine Protected Areas dataset (MPAS-mine.csv) and summarize the number of Marine Protected Areas by United Nations Geoscheme Region (_UN Geoscheme_).
]

---

.large[
- Import the Marine Protected Areas dataset (MPAS-your.csv), and summarize the number of Marine Protected Areas by country (_Country full_).
]

---

class: center, middle, dk-section-title
background-image:url("images/hans-heiner-buhr-OKe4Q8azVNU-unsplash.jpg")
background-size: cover
# Missing, Implicit, or Misplaced Grouping Variables

???

When we learned about data organization earlier, I mentioned that grouping variables should preferably be in their own column. Now, we'll see what can be done when we get data with missing, implicit, or misplaced grouping variable.

---

### Missing ... 🤷

???

First off, if the grouping variables are missing altogether from the data, there's not much we can do.

---

### Implicit or misplaced

???

But if they are somewhere in the data, just not where they're supposed to be, we have more options.

A common issue that affects grouping variables is when they get tangled up inside existing columns.

???

either embedded in the data as subheaders

--
.large[
- Part of a compound value
]

???

or misplaced inside an existing variable but as part of a compound value. We'll deal with compound values in the lext lesson, but for now lets look at grouping values embedded as subheaders.

---

## Embedded subheaders

.left-column[
**Hot drinks**
Coffee  
Tea  
Hot chocolate
**Cold Drinks**
Soda  
Juice  
Water  
Beer  
]

???
We can see this excerpt from a cafeteria menu as an example of embedded subheaders. Why is that?

- Rows that correspond to values in grouping variables

- Embedded in the data rectangle instead of in their own column

- Human-readable, but hard to work with (for example: calculating group-wise summaries)

]

???

Because we have rows that correspond to values in a grouping variable. 
In this case, the text in bold group the menu items into hot or cold drinks, but these values are entangled with the actual items. This is very-human readable, but it can be hard to work with.

---

.panelset[
.panel[.panel-name[Menu]
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#xjsygdwdwd .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #f4f4f9;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#xjsygdwdwd .gt_heading {
  background-color: #f4f4f9;
  text-align: center;
  border-bottom-color: #f4f4f9;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#xjsygdwdwd .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #f4f4f9;
  border-bottom-width: 0;
}

#xjsygdwdwd .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #f4f4f9;
  border-top-width: 0;
}

#xjsygdwdwd .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#xjsygdwdwd .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#xjsygdwdwd .gt_col_heading {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#xjsygdwdwd .gt_column_spanner_outer {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#xjsygdwdwd .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#xjsygdwdwd .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#xjsygdwdwd .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#xjsygdwdwd .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#xjsygdwdwd .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#xjsygdwdwd .gt_from_md > :first-child {
  margin-top: 0;
}

#xjsygdwdwd .gt_from_md > :last-child {
  margin-bottom: 0;
}

#xjsygdwdwd .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#xjsygdwdwd .gt_stub {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#xjsygdwdwd .gt_summary_row {
  color: #333333;
  background-color: #f4f4f9;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#xjsygdwdwd .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#xjsygdwdwd .gt_grand_summary_row {
  color: #333333;
  background-color: #f4f4f9;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#xjsygdwdwd .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#xjsygdwdwd .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#xjsygdwdwd .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#xjsygdwdwd .gt_footnotes {
  color: #333333;
  background-color: #f4f4f9;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#xjsygdwdwd .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#xjsygdwdwd .gt_sourcenotes {
  color: #333333;
  background-color: #f4f4f9;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#xjsygdwdwd .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#xjsygdwdwd .gt_left {
  text-align: left;
}

#xjsygdwdwd .gt_center {
  text-align: center;
}

#xjsygdwdwd .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#xjsygdwdwd .gt_font_normal {
  font-weight: normal;
}

#xjsygdwdwd .gt_font_bold {
  font-weight: bold;
}

#xjsygdwdwd .gt_font_italic {
  font-style: italic;
}

#xjsygdwdwd .gt_super {
  font-size: 65%;
}

#xjsygdwdwd .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="xjsygdwdwd" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 25px;">Item</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 25px;">Price</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left" style="font-size: 25px;">Hot Drinks</td>
<td class="gt_row gt_right" style="font-size: 25px;">NA</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 25px;">Coffee</td>
<td class="gt_row gt_right" style="font-size: 25px;">12</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 25px;">Tea</td>
<td class="gt_row gt_right" style="font-size: 25px;">9</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 25px;">Hot chocolate</td>
<td class="gt_row gt_right" style="font-size: 25px;">9</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 25px;">Cold Drinks</td>
<td class="gt_row gt_right" style="font-size: 25px;">NA</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 25px;">Soda</td>
<td class="gt_row gt_right" style="font-size: 25px;">10</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 25px;">Juice</td>
<td class="gt_row gt_right" style="font-size: 25px;">13</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 25px;">Water</td>
<td class="gt_row gt_right" style="font-size: 25px;">8</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 25px;">Beer</td>
<td class="gt_row gt_right" style="font-size: 25px;">8</td></tr>
  </tbody>
  
  
</table></div>
]

```r
cafeteria <-
  tibble::tribble(
    ~Item, ~Price,
    "Hot Drinks", NA,
    "Coffee", 12L,
    "Tea", 9L,
    "Hot chocolate", 9L,
    "Cold Drinks", NA,
    "Soda", 10L,
    "Juice", 13L,
    "Water", 8L,
    "Beer", 8L
  )
```
]
]

???

Here's the cafeteria menu again, this time as a tibble object and with prices for each item.

First up, notice the grouping variables that appear as subheaders inside the item column. Good data organization calls for having this information in a separate variable, in this case, one that groups drinks by type.

---

## Tidying embedded subheaders
.large[
.b.purple[`untangle2()`] from 📦 .rrured.b[`unheadr`]
]

???

Until recently, there was no easy way to handle embedded subheaders, but now we can, using the untangle2 function from the unheadr package. Untangle2 will work in most cases, with two main approaches.

1. Match with a .orange.b[regular expression]

???

First, we can put these grouping information into its own variable if the embedded subheaders share a prefix or suffix or anything in common that can be matched with a regular expression.

2. Position (sometimes) identifiable as the rows with NAs in all the variables except for the one with the subheaders

???

If not, we can often figure out which rows have embedded subheaders, because the rows they consume inside other variables tend to be otherwise empty.

---

## `untangle2()`

```
## # A tibble: 9 x 2
##   Item          Price
##   <chr>         <int>
## 1 Hot Drinks       NA
## 2 Coffee           12
## 3 Tea               9
## 4 Hot chocolate     9
## 5 Cold Drinks      NA
## 6 Soda             10
## 7 Juice            13
## 8 Water             8
## 9 Beer              8
```

???

We can see that both of these approaches would work in our cafeteria data.
This is because the embedded values use to group the hot drinks and the cold drinks have a word that appears in both at the same position. And also, there is nothing else in the rows with these values.

```r
untangle2(
  df,
  regex, # regular expression to match subheaders
  orig, # variable with the subheaders
  new
) # name of the new variable with the group values
```

???

The untangle2 function also works like a dplyr function. It works well with pipes and has a similar syntax.

The main arguments are our data object, a regular expression to match the subheaders, the name of the variable with the subheaders, and a name for the new grouping variable.

---

## `untangle2()`

Match using .orange.b[regex]

```r
cafeteria %>%
  untangle2("Drinks$", Item, drink_type) %>% 
  clean_names()
```

```
## # A tibble: 7 x 3
##   item          price drink_type 
##   <chr>         <int> <chr>      
## 1 Coffee           12 Hot Drinks 
## 2 Tea               9 Hot Drinks 
## 3 Hot chocolate     9 Hot Drinks 
## 4 Soda             10 Cold Drinks
## 5 Juice            13 Cold Drinks
## 6 Water             8 Cold Drinks
## 7 Beer              8 Cold Drinks
```

???

Lets match the grouping values, or subheaders, with a regular expression.
Here we pipe the cafeteria object into untangle2, and the regular expression matches the word drinks with a capital D at the end of a string. The column to work on is the item column, and we'll name the new grouping variable drink underscore type.

Looking at the result, the function did its job, putting the grouping values in a separate column with a group for each observation. The original rows with the subheaders were removed. These data are now usable.

---

## `untangle2()`

.orange.b[Regexp] match and summarize

```r
cafeteria %>%
  untangle2("Drinks$", Item, drink_type) %>%
  clean_names() %>%
  group_by(drink_type) %>%
  summarize(mean_price = mean(price))
```

```
## # A tibble: 2 x 2
##   drink_type  mean_price
## * <chr>            <dbl>
## 1 Cold Drinks       9.75
## 2 Hot Drinks       10
```

???
With a tidy version of the same data, we can use dplyr and perform calculations on groups. Here, we group the data by drink type and get the mean price for the cold and the hot drinks. This wouldn't be possible at all with the data in their previous presentation.

---

### Alternation

???

If we cant match the embedded grouping values with a regular expression, but there aren't that many of them and we can recognize them somehow, we can use alternation and pass the series of alternatives to the untangle2 function.

```r
cafeteria %>%
  untangle2("Hot Drinks|Cold Drinks", Item, drink_type) %>%
  clean_names()
```

???

See? we passed the possible alternatives for matching, spelled out in full, and got the same results as before.

---

### Alternation

```r
cafeteria %>%
  untangle2("Hot Drinks|Cold Drinks", Item, drink_type) %>%
  clean_names() %>%
  group_by(drink_type) %>%
  summarize(mean_price = mean(price))
```

```
## # A tibble: 2 x 2
##   drink_type  mean_price
## * <chr>            <dbl>
## 1 Cold Drinks       9.75
## 2 Hot Drinks       10
```

???

And now we repeat the calculations on groups, untangling the groups using alternation and getting the mean price for each type of drink.

---

### Identify subheaders by context (`NA` values)

???

This third option is more involved, but it can help us when there are many distinct embedded grouping values and they share nothing in common. When we can't match them with a regular expression and it becomes impractical to locate and write them all for alternation, we can identify these subheaders by context.

The context here would be those values in our column of interest in which all other rows are empty, because these embedded subheaders are used to imply that everything below them belongs to a group.

```r
cafeteria %>%
  filter(across(-Item, is.na)) %>%
  select(Item) %>%
  mutate(tag = paste0("subheader_", Item)) %>%
  right_join(cafeteria) %>%
  untangle2("^subheader", tag, drink_type) %>%
  select(-tag) %>%
  mutate(drink_type = stringr::str_remove(drink_type, "^subheader_"))
```

```
## # A tibble: 7 x 3
##   Item          Price drink_type 
##   <chr>         <int> <chr>      
## 1 Coffee           12 Cold Drinks
## 2 Tea               9 Cold Drinks
## 3 Hot chocolate     9 Cold Drinks
## 4 Soda             10 Cold Drinks
## 5 Juice            13 Cold Drinks
## 6 Water             8 Cold Drinks
## 7 Beer              8 Cold Drinks
```

???

This sequence will give us the same result as the ones before. It looks complicated, but it is just a series of dplyr operation to find the rows with the group values, tag them, and match them.

---

```r
cafeteria %>%
  filter(across(-Item, is.na)) %>%
  select(Item) %>%
  mutate(tag = paste0("subheader_", Item))
```

```
## # A tibble: 2 x 2
##   Item        tag                  
##   <chr>       <chr>                
## 1 Hot Drinks  subheader_Hot Drinks 
## 2 Cold Drinks subheader_Cold Drinks
```

???

Lets break this into two main step so I can explain this better.

Filter rows with NAs in all columns except the one holding the subheaders

???

The first thing we do is to filter the data and keep the rows that have NA values in all the columns, except the one with the subheaders.

Keep only the column with subheaders

???

Then we only keep this focal column,

Create a new variable that includes a pattern that can be matched with .orange.b[regex]

???

And use mutate to make a copy of its values, but adding a tag to the strings that will let us match them with regex

---

```r
... %>%
  right_join(cafeteria) %>%
  untangle2("^subheader", tag, drink_type) %>%
  select(-tag) %>%
  mutate(drink_type = stringr::str_remove(drink_type, "^subheader_"))
```

???

Once we have this object with the tagged group values, we can proceed

Join resulting object with original data

???

Next, we'll join this object with the original data, in this case we're doing a right join usin dplyr. This join will merge the two objects, so that the group values in the original data will be labelled with something of our choosing and ready for matching.

Put implicit grouping variable into its own column

???

After this, we proceed just like before, matching the prefix we just added, and untangling the data.

Clean up resulting data

???

After that, we can clean up the data by removing the prefix we used for tagging the subheaders. In general, I believe that this approach can be quite useful for large datasets with many subheaders that don't share a common pattern but all appear in otherwise empty rows.

---

class: my-turn
# My turn
.large[
- Load the `primates2017` dataset bundled with 📦 `unheadr` and create a new column that groups the different species by geographic region. 
]

---

class: inverse
# Your turn
.large[
- Load the `primates2017` dataset bundled with 📦 `unheadr` and create a new column that groups the different species by taxonomic family.  
]

> In biology, taxonomic families all end in the suffix "_DAE_"

---

class: center, middle, dk-section-title
background-image:url("images/max-vertsanov-qvRuue12Huw-unsplash.jpg")
background-size: fill
# Compound Values

???

Another common issue that should be overlooked is when we have compound values in our data

---

## Compound values

???

In the context of tidy data, remember that each cell in a data rectangle should have only value. If there are two or more values from different variables or observations in the same cell, then we have compound values.

???

If this happens, then by definition the data are not tidy.

???

More importantly, if we don't separate compound values, parts of of these values may go overlooked and important information may be lost. We'll see how this can happen in the upcoming example.

---

### Tidying compound values

???

We have some options if we want to tidy up data with compound values. Existing functions from tidyverse packages can help us with this.

**`separate()`** columns

**`separate_rows()`**
]

???

First is to separate columns or rows. When there are compound values in a cell, there is often a character used to separate each value. If so, we can identify and use these separators to break up our compound values. For this we use functions from tidyr, one for columns and one for rows.

**`str_extract()`** substrings into new columns
]

???

Another possibility is to use regular expressions and use the stringr package to match part of our compound values and put the matched content into a new variable.

Lets dive into the example.

---

.panelset[
.panel[.panel-name[households]
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#iiglbbwuva .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#iiglbbwuva .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#iiglbbwuva .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#iiglbbwuva .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#iiglbbwuva .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#iiglbbwuva .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#iiglbbwuva .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#iiglbbwuva .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#iiglbbwuva .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#iiglbbwuva .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#iiglbbwuva .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#iiglbbwuva .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#iiglbbwuva .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#iiglbbwuva .gt_from_md > :first-child {
  margin-top: 0;
}

#iiglbbwuva .gt_from_md > :last-child {
  margin-bottom: 0;
}

#iiglbbwuva .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#iiglbbwuva .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#iiglbbwuva .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#iiglbbwuva .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#iiglbbwuva .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#iiglbbwuva .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#iiglbbwuva .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#iiglbbwuva .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#iiglbbwuva .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#iiglbbwuva .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#iiglbbwuva .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#iiglbbwuva .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#iiglbbwuva .gt_left {
  text-align: left;
}

#iiglbbwuva .gt_center {
  text-align: center;
}

#iiglbbwuva .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#iiglbbwuva .gt_font_normal {
  font-weight: normal;
}

#iiglbbwuva .gt_font_bold {
  font-weight: bold;
}

#iiglbbwuva .gt_font_italic {
  font-style: italic;
}

#iiglbbwuva .gt_super {
  font-size: 65%;
}

#iiglbbwuva .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="iiglbbwuva" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">Participant.ID</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">Education</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">Residence</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1">Bonus</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left">GHC21</td>
<td class="gt_row gt_left">Vocational</td>
<td class="gt_row gt_left">Living Alone-Cat</td>
<td class="gt_row gt_right">500</td></tr>
    <tr><td class="gt_row gt_left">MYL11</td>
<td class="gt_row gt_left">High School</td>
<td class="gt_row gt_left">Family Home-Cat</td>
<td class="gt_row gt_right">400</td></tr>
    <tr><td class="gt_row gt_left">LLB16</td>
<td class="gt_row gt_left">Graduate</td>
<td class="gt_row gt_left">Family Home-Dog</td>
<td class="gt_row gt_right">400</td></tr>
    <tr><td class="gt_row gt_left">AAH08</td>
<td class="gt_row gt_left">Vocational</td>
<td class="gt_row gt_left">Shared Housing-None</td>
<td class="gt_row gt_right">450</td></tr>
    <tr><td class="gt_row gt_left">PCG91</td>
<td class="gt_row gt_left">Vocational</td>
<td class="gt_row gt_left">Family Home-Other</td>
<td class="gt_row gt_right">500</td></tr>
    <tr><td class="gt_row gt_left">ACC22, PMM02</td>
<td class="gt_row gt_left">High School</td>
<td class="gt_row gt_left">Shared Housing-None</td>
<td class="gt_row gt_right">400</td></tr>
    <tr><td class="gt_row gt_left">MJM13</td>
<td class="gt_row gt_left">Postgraduate</td>
<td class="gt_row gt_left">Living Alone-Dog</td>
<td class="gt_row gt_right">500</td></tr>
  </tbody>
  
  
</table></div>
]

```r
households <-
  tibble::tribble(
    ~Participant.ID, ~Education, ~Residence, ~Bonus,
    "GHC21", "Vocational", "Living Alone-Cat", 500L,
    "MYL11", "High School", "Family Home-Cat", 400L,
    "LLB16", "Graduate", "Family Home-Dog", 400L,
    "AAH08", "Vocational", "Shared Housing-None", 450L,
    "PCG91", "Vocational", "Family Home-Other", 500L,
    "ACC22, PMM02", "High School", "Shared Housing-None", 400L,
    "MJM13", "Postgraduate", "Living Alone-Dog", 500L
  )
```
]
]

???

This object called households has some demographic data about fictional households, and it has compound values.

First, look closely at the participant ID variable. There is one instance with two household IDS, separated by a comma. In this case we might assume that this means that the values for the rest of the variables are the same for these two households.

Then, notice the Residence column, it has values for the type of residence, but there is also data on pets in the same cells, separated by a dash.

Lets see how we can sort out these issues.

---

## Separate Residence and 'Pet' variables

???

First, lets separate the residence and Pet variables, which should be in two columns for more usability.

???

For this, we'll use separate from tidyr. this function splits a character column into multiple columns.

_Arguments_

???

To do so, we tell the function what is being used to separate values with the argument sep.

And with the into argument, we pass a character vector with the names of the new columns to create.

---

## Separate Residence and 'Pet' variables

```r
households %>%
  separate(
    col = Residence, # columns to separate
    into = c("Residence", "Pet"), # names of new variables to create
    sep = "-"
  ) # separator between columns
```

```
## # A tibble: 7 x 5
##   Participant.ID Education    Residence      Pet   Bonus
##   <chr>          <chr>        <chr>          <chr> <int>
## 1 GHC21          Vocational   Living Alone   Cat     500
## 2 MYL11          High School  Family Home    Cat     400
## 3 LLB16          Graduate     Family Home    Dog     400
## 4 AAH08          Vocational   Shared Housing None    450
## 5 PCG91          Vocational   Family Home    Other   500
## 6 ACC22, PMM02   High School  Shared Housing None    400
## 7 MJM13          Postgraduate Living Alone   Dog     500
```

???
Lets see. 
We pipe the households object into separate, specify that we want to separate the Residence column, into two columns, residence and pet, and a dash is being used to separate values.

The resulting object looks better, and now we have pets in its own column.

---

### Match and extract

???

Now let me demonstrate the other approach, in which we match and extract a substring and put it into a separate column.

```r
households %>%
  mutate(Pet = str_extract(Residence, "\\b\\w+$"))
```

```
## # A tibble: 7 x 5
##   Participant.ID Education    Residence           Bonus Pet  
##   <chr>          <chr>        <chr>               <int> <chr>
## 1 GHC21          Vocational   Living Alone-Cat      500 Cat  
## 2 MYL11          High School  Family Home-Cat       400 Cat  
## 3 LLB16          Graduate     Family Home-Dog       400 Dog  
## 4 AAH08          Vocational   Shared Housing-None   450 None 
## 5 PCG91          Vocational   Family Home-Other     500 Other
## 6 ACC22, PMM02   High School  Shared Housing-None   400 None 
## 7 MJM13          Postgraduate Living Alone-Dog      500 Dog
```

???
For this, we use the str extract function from stringr inside mutate to match a pattern in the residence column. The regular expression uses a word boundary, the shorthand sequence for word characters, a quantifier, and an end anchor to tell the regex engine the match the last word in a string. In this case, the pets are the last words in the string, so these matches will be extracted into a new Pet column.

After this, we might want to clean up the original residence column, which we could easily do with str remove. This approach can be very useful if the compound values don't have a clear separator and instead we can match parts of the values with regular expressions.

---

## Separate Participant IDs

???

Now, lets see what we can do when the compound values are for values meant to be across rows and not columns.

.large[
.b.purple[`separate_rows()`] from 📦 .rrured.b[`tidyr`] 
- Separates multiple delimited observations within a column and places each one in its own row.]

???

To separat rows, we use the separate rows function from tidyr, what this does it to separate multiple observations in a column into two or more rows.

---

## Separate Participant IDs

```r
households %>%
  separate_rows(Participant.ID,
    sep = ", ")
```

```
## # A tibble: 8 x 4
##   Participant.ID Education    Residence           Bonus
##   <chr>          <chr>        <chr>               <int>
## 1 GHC21          Vocational   Living Alone-Cat      500
## 2 MYL11          High School  Family Home-Cat       400
## 3 LLB16          Graduate     Family Home-Dog       400
## 4 AAH08          Vocational   Shared Housing-None   450
## 5 PCG91          Vocational   Family Home-Other     500
## 6 ACC22          High School  Shared Housing-None   400
## 7 PMM02          High School  Shared Housing-None   400
## 8 MJM13          Postgraduate Living Alone-Dog      500
```

???

The syntax for this function is quite simple, we pipe the households object into separate rows, then specify which column has values that need to be separated, and what the separator is.
In this case, we want to split participant ids, and the separator is a comma followed by a space.

In the result, we have one more row because the compound value was split across rows, and the values in the other columns were taken to be the same for the two values that were together in the first place.

---

.large[
- Import the Marine Protected Areas dataset (MPAS-mine.csv), separate the country codes variable (ISO3 and UN scheme) into columns and the Reference variable into rows.]

---

.large[
- Import the Marine Protected Areas dataset (MPAS-your.csv), separate the country codes variable (ISO3 and UN scheme) into columns and the Reference variable into rows.]

> Keep an eye on the separators

---

class: center, middle, dk-section-title
background-image:url("images/armelle-danjour-kwGqIuNrM5E-unsplash.jpg")
background-size: cover
# Duplicates

???

At this point, we can discuss the very common issue of duplicate values in our data.

Before we continue, I'd like to define duplication in the context of data cleaning.
Typically, when we have two or more copies of the same record, observation, or data point, these can be said to be duplicated.

However, it's important to know that defining duplication in our data is up to us. This will become evident in the examples.

---

## Problems with duplicates

???

In this section we'll learn how to handle duplicates, but first let's see what can happen if we leave them in our data.
--

???

First, duplicate values bloat our data, and in larger datasets they can easily get out of hand if we somehow end up duplicating the duplicates. This has implication for file size, efficiency, and readability.

???

Also, unintentional repetition can be costly, not just in terms of computing and storage, but also in real applications. For example, if a company has duplicated records for their customers, they may end up unintentionally shipping a product or sample more than once to the same customer.

???

Another major complication that comes from duplicate data, is that we can end up with results and data that are downright incorrect or biased. The most obvious issue would be double counting or inflating our numbers without realizing.

---

## What can we do?

???

To deal with duplication, we can again call on a number of packages with purpose-built tools.

???

To fix duplication, we first need to identify it. For this, we can use the get dupes function from the janitor package.

???

If we find duplicates in our data, and we don't want them, then we can use the distinct function from dplyr to get rid of them.

---

## Duplicated values

???

As I mentioned earlier, it is up to us to define duplication in our data.

???

The most common usage, is when duplicates refer to identical copies of the same row.

???

But we can also define duplicates in which not all values are identical across variables. A common practice is to pick the variable or set of variables that define our observational units and look for duplication here, regardless of what's in the other columns.

???

And, it's also possible to define duplicates in any arbitrary set of variables.

---

.panelset[
.panel[.panel-name[pizza_orders]
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#eosamumnia .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#eosamumnia .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#eosamumnia .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#eosamumnia .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#eosamumnia .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#eosamumnia .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#eosamumnia .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#eosamumnia .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#eosamumnia .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#eosamumnia .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#eosamumnia .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#eosamumnia .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#eosamumnia .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#eosamumnia .gt_from_md > :first-child {
  margin-top: 0;
}

#eosamumnia .gt_from_md > :last-child {
  margin-bottom: 0;
}

#eosamumnia .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#eosamumnia .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#eosamumnia .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#eosamumnia .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#eosamumnia .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#eosamumnia .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#eosamumnia .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#eosamumnia .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#eosamumnia .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#eosamumnia .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#eosamumnia .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#eosamumnia .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#eosamumnia .gt_left {
  text-align: left;
}

#eosamumnia .gt_center {
  text-align: center;
}

#eosamumnia .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#eosamumnia .gt_font_normal {
  font-weight: normal;
}

#eosamumnia .gt_font_bold {
  font-weight: bold;
}

#eosamumnia .gt_font_italic {
  font-style: italic;
}

#eosamumnia .gt_super {
  font-size: 65%;
}

#eosamumnia .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="eosamumnia" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 26px;">CustomerID</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 26px;">Address</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 26px;">City</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 26px;">State</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left" style="font-size: 26px;">Newman</td>
<td class="gt_row gt_left" style="font-size: 26px;">Apartment 5E, 129 West 81st Street</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York City</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 26px;">susan_A</td>
<td class="gt_row gt_left" style="font-size: 26px;">185 West 74th Street</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York City</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 26px;">susan_A</td>
<td class="gt_row gt_left" style="font-size: 26px;">185 West 74th Street</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York City</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 26px;">js1994</td>
<td class="gt_row gt_left" style="font-size: 26px;">Apartment 5A, 129 West 81st Street</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York City</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 26px;">Eric</td>
<td class="gt_row gt_left" style="font-size: 26px;">Unit 1B, 4186 Willoughby Avenue</td>
<td class="gt_row gt_left" style="font-size: 26px;">Brooklyn</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 26px;">Dash</td>
<td class="gt_row gt_left" style="font-size: 26px;">Unit 1B, 4186 Willoughby Avenue</td>
<td class="gt_row gt_left" style="font-size: 26px;">Brooklyn</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 26px;">Rakeem</td>
<td class="gt_row gt_left" style="font-size: 26px;">Unit 1B, 4186 Willoughby Avenue</td>
<td class="gt_row gt_left" style="font-size: 26px;">Brooklyn</td>
<td class="gt_row gt_left" style="font-size: 26px;">New York</td></tr>
  </tbody>
  
  
</table></div>
]

```r
pizza_orders <-
  tibble::tribble(
    ~CustomerID, ~Address, ~City, ~State,
    "Newman", "Apartment 5E, 129 West 81st Street", "New York City", "New York",
    "susan_A", "185 West 74th Street", "New York City", "New York",
    "susan_A", "185 West 74th Street", "New York City", "New York",
    "js1994", "Apartment 5A, 129 West 81st Street", "New York City", "New York",
    "Eric", "Unit 1B, 4186 Willoughby Avenue", "Brooklyn", "New York",
    "Dash", "Unit 1B, 4186 Willoughby Avenue", "Brooklyn", "New York",
    "Rakeem", "Unit 1B, 4186 Willoughby Avenue", "Brooklyn", "New York"
  )
```
]
]

???

Let's look at duplicates in an example dataset.
This object called pizza orders records customers that ordered pizza from a restaurant.

This is a small dataset, so we can see straight away that there are two rows with the exact same values across all columns. These are duplicates by definition, but we could also define duplication at other levels, for example, duplicates of the same address or city.

---

???

Let's identify the duplicate records using the janitor package.

```r
pizza_orders %>%
  get_dupes() # all variables by default
```

```
## # A tibble: 2 x 5
##   CustomerID Address              City          State    dupe_count
##   <chr>      <chr>                <chr>         <chr>         <int>
## 1 susan_A    185 West 74th Street New York City New York          2
## 2 susan_A    185 West 74th Street New York City New York          2
```

* Adds a `dupe_count` variable showing the number of rows sharing the duplicated values  
* Puts the input variables at the beginning of the resulting data frame

???

We simply pass the pizza orders object to get dupes. Without any other arguments, the function will work on all variables. The result will be a table with only the duplicate records, with a new column that tells us how many copies there were.

---

???

Let's repeat the operation, but this time we can specify a variable, by name, in which we want to look for duplicates. In this case, let's see if any customer placed more than one order.

```r
pizza_orders %>%
  get_dupes(CustomerID)
```

```
## # A tibble: 2 x 5
##   CustomerID dupe_count Address              City          State   
##   <chr>           <int> <chr>                <chr>         <chr>   
## 1 susan_A             2 185 West 74th Street New York City New York
## 2 susan_A             2 185 West 74th Street New York City New York
```

???

Coincidentally, we get the same result as before, but this is not always the case.

---

???

Now, let's look for duplicates in an arbitrary combination of variables. Let's say we want to find duplicates in the combination of address and city. This implies that we don't care if the customers are different.

> now accepts 📦 .rrured.b[`tidyselect`] helpers

```r
pizza_orders %>%
  get_dupes(Address, starts_with("Cit"))
```

```
## # A tibble: 5 x 5
##   Address                         City          dupe_count CustomerID State   
##   <chr>                           <chr>              <int> <chr>      <chr>   
## 1 185 West 74th Street            New York City          2 susan_A    New York
## 2 185 West 74th Street            New York City          2 susan_A    New York
## 3 Unit 1B, 4186 Willoughby Avenue Brooklyn               3 Eric       New York
## 4 Unit 1B, 4186 Willoughby Avenue Brooklyn               3 Dash       New York
## 5 Unit 1B, 4186 Willoughby Avenue Brooklyn               3 Rakeem     New York
```

???

To do this, we simply pass the variable names, separated by a comma. Notice that we can use helper functions from tidyselect to match variable names. In the result, we see the two addresses that placed more than one order, even when they were placed by different customers.

---

???

Now that we identified duplicate records, we can remove them if we want. For this, we use the distinct function from dplyr, which does essentially the opposite of get dupes.

```r
pizza_orders %>%
  distinct() # all variables by default
```

```
## # A tibble: 6 x 4
##   CustomerID Address                            City          State   
##   <chr>      <chr>                              <chr>         <chr>   
## 1 Newman     Apartment 5E, 129 West 81st Street New York City New York
## 2 susan_A    185 West 74th Street               New York City New York
## 3 js1994     Apartment 5A, 129 West 81st Street New York City New York
## 4 Eric       Unit 1B, 4186 Willoughby Avenue    Brooklyn      New York
## 5 Dash       Unit 1B, 4186 Willoughby Avenue    Brooklyn      New York
## 6 Rakeem     Unit 1B, 4186 Willoughby Avenue    Brooklyn      New York
```

???

If we use the distinct function without any arguments, it will work on all the variables, and drop the duplicated record we had seen earlier.

---

???

If we do the same for the customer ID variable, each unique value for this column will appear only once, regardless of what's in the other columns.

```r
pizza_orders %>%
  distinct(CustomerID, .keep_all = TRUE)
```

???

The resulting data has no more duplicated entries for customer ID

> Use `.keep_all = FALSE` (default) to drop all other variables

???

Notice, that distinct has a useful argument for either keeping or dropping all the other variables.

---

???

Let's try again with an arbitrary selection of variables, just we did before. We want the unique combinations of addresses and cities.

>  📦 .b.rrured[`tidyselect`] helpers enabled by using .b[`across`] semantics

```r
pizza_orders %>%
  distinct(across(c(Address, starts_with("Cit"))))
```

```
## # A tibble: 4 x 2
##   Address                            City         
##   <chr>                              <chr>        
## 1 Apartment 5E, 129 West 81st Street New York City
## 2 185 West 74th Street               New York City
## 3 Apartment 5A, 129 West 81st Street New York City
## 4 Unit 1B, 4186 Willoughby Avenue    Brooklyn
```

???

To use tidyselect like we did with get dupes, we use the across function. We pass the address and part of cities. In the result, the default behaviour of distinct dropped the rest of the colums and now we only have all the unique combinations of address and city.

---

>⚠️ If `.keep_all = TRUE` and there are duplicates in other variables, `distinct` only keeps the first row

```r
pizza_orders %>%
  distinct(across(c(Address, starts_with("City"))), .keep_all = TRUE)
```

```
## # A tibble: 4 x 4
##   CustomerID Address                            City          State   
##   <chr>      <chr>                              <chr>         <chr>   
## 1 Newman     Apartment 5E, 129 West 81st Street New York City New York
## 2 susan_A    185 West 74th Street               New York City New York
## 3 js1994     Apartment 5A, 129 West 81st Street New York City New York
## 4 Eric       Unit 1B, 4186 Willoughby Avenue    Brooklyn      New York
```

???

This is the same as before, but we used the keep all argument to keep the rest of the columns. Be aware of the default behavior here, because distinct will keep the first row even if there are duplicates in other variables outside of our selection.

---

# My turn

.large[
- Load the messy Age of Empires units dataset bundled with `unheadr` (AOEunits_raw) and discard units that are not of Type "Archer".
]

---

.large[
- Load the messy Age of Empires units dataset bundled with `unheadr` (AOEunits_raw) and keep only units of Type "Cavalry".
]

---

class: center, middle, dk-section-title
background-image:url("images/jonathan-safa-YcxOAC5DpDA-unsplash.jpg")
background-size: cover
# Broken Values

???

We talked about broken headers earlier, but other values in our data can also be broken, which can really slow us down.

---

# Broken values

???

For this section, broken values refers to instances where values are broken across rows. This is often done to save horizontal space in documents.

???

When this happens, Na or blank values can end up padding the empty spaces and this is not good for data organization.

???

Broken values are particularly problematic when either grouping variables or the variables with the observational units are broken across rows.

---

.panelset[
.panel[.panel-name[olympics]
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#dicvfezwvz .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#dicvfezwvz .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#dicvfezwvz .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#dicvfezwvz .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#dicvfezwvz .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#dicvfezwvz .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#dicvfezwvz .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#dicvfezwvz .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#dicvfezwvz .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#dicvfezwvz .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#dicvfezwvz .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#dicvfezwvz .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#dicvfezwvz .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#dicvfezwvz .gt_from_md > :first-child {
  margin-top: 0;
}

#dicvfezwvz .gt_from_md > :last-child {
  margin-bottom: 0;
}

#dicvfezwvz .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#dicvfezwvz .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#dicvfezwvz .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#dicvfezwvz .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#dicvfezwvz .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#dicvfezwvz .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#dicvfezwvz .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#dicvfezwvz .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#dicvfezwvz .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#dicvfezwvz .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#dicvfezwvz .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#dicvfezwvz .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#dicvfezwvz .gt_left {
  text-align: left;
}

#dicvfezwvz .gt_center {
  text-align: center;
}

#dicvfezwvz .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#dicvfezwvz .gt_font_normal {
  font-weight: normal;
}

#dicvfezwvz .gt_font_bold {
  font-weight: bold;
}

#dicvfezwvz .gt_font_italic {
  font-style: italic;
}

#dicvfezwvz .gt_super {
  font-size: 65%;
}

#dicvfezwvz .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="dicvfezwvz" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 19px;">Edition</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 19px;">Country</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 19px;">Soccer.gold.medal.(men)</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 19px;">Wrestling.gold.medal.(men.middleweight)</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">weight class limit:</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">82 kg</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 19px;">Los Angeles 1984</td>
<td class="gt_row gt_left" style="font-size: 19px;">USA</td>
<td class="gt_row gt_left" style="font-size: 19px;">France</td>
<td class="gt_row gt_left" style="font-size: 19px;">USA</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 19px;">Barcelona</td>
<td class="gt_row gt_left" style="font-size: 19px;">Spain</td>
<td class="gt_row gt_left" style="font-size: 19px;">Spain</td>
<td class="gt_row gt_left" style="font-size: 19px;">USA</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 19px;">1992</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 19px;">Atlanta 1996</td>
<td class="gt_row gt_left" style="font-size: 19px;">USA</td>
<td class="gt_row gt_left" style="font-size: 19px;">Nigeria</td>
<td class="gt_row gt_left" style="font-size: 19px;">Russia</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">weight class limit:</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">85 kg</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 19px;">Sydney 2000</td>
<td class="gt_row gt_left" style="font-size: 19px;">Australia</td>
<td class="gt_row gt_left" style="font-size: 19px;">Cameroon</td>
<td class="gt_row gt_left" style="font-size: 19px;">Russia</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 19px;">London</td>
<td class="gt_row gt_left" style="font-size: 19px;">UK</td>
<td class="gt_row gt_left" style="font-size: 19px;">Mexico</td>
<td class="gt_row gt_left" style="font-size: 19px;">Azerbaijan</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 19px;">2012</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td>
<td class="gt_row gt_left" style="font-size: 19px;">NA</td></tr>
  </tbody>
  
  
</table></div>
]

```r
olympics <-
  tibble::tribble(
    ~Edition, ~Country, ~`Soccer.gold.medal.(men)`, ~`Wrestling.gold.medal.(men.middleweight)`,
    NA, NA, NA, "weight class limit:",
    NA, NA, NA, "82 kg",
    "Los Angeles 1984", "USA", "France", "USA",
    "Barcelona", "Spain", "Spain", "USA",
    "1992", NA, NA, NA,
    "Atlanta 1996", "USA", "Nigeria", "Russia",
    NA, NA, NA, "weight class limit:",
    NA, NA, NA, "85 kg",
    "Sydney 2000", "Australia", "Cameroon", "Russia",
    "London", "UK", "Mexico", "Azerbaijan",
    "2012", NA, NA, NA
  )
```
]
]

???

These are our example data for broken values. These are data show gold medals in different olympic games for two sports. These data are quite messy, starting with the column names, but we can fix this easily. Next, see how the rows are full of NAs. This is because there are broken values in the data.

Looking closely at the edition column, some of the values are split up across two consecutive rows, maybe to make the data fit better on a page. This happened for the Barcelona 1992 and London 2012 games. The year is in the next row. What this ends up causing is confusion and lots of NAs. Another column with issues is the last one, wrestling gold medal. In this one, for some reason, information about the changes to the weight class appear in the data, and the values are also broken. Let's clean this up.

---

## 'Unbreaking' values

???

To clean broken values, we can also use some existing tools, but they have some limitations.

???

The main one being that we can mostly just unbreak values when they are split up across two consecutive rows. Any more than that and the process becomes unmanageable.

.large[
Match the trailing or leading half of the value with a .orange.b[regexp] and 📦 .rrured.b[`unheadr`]
]

???

To unbreak values, we can use functions from unheadr, and this approach uses regular expressions to match either the leading or the trailing half of the broken values.

---

## 'Unbreaking' values 
### Leading

.pull-right[
**Retriever**  
(flat-coated)  
Bulldog (American)   
Bullmastiff  
**Retriever**  
(golden)  
Poodle  
]

???

This is what we mean by the leading half of a broken value.
This simple example has some observations, for restaurants and for dog breeds. I highlighted here the leading half, like Fast in fast food. In this case with the restaurants, these leading parts start with an uppercase letter, unlike the remaining half of the value.

---

## 'Unbreaking' values
### Trailing

.pull-right[
Retriever  
**(flat-coated)**  
Bulldog (American)   
Bullmastiff  
Retriever  
**(golden)**  
Poodle  
]

???

Trailing halves are the second of two parts of a value. Like food in fast food. This we can recognize because it starts with a lowercase letter. For the dog breeds, some annotations in parentheses got broken up and we can recognize them because these trailing halves start with an opening brace.

---

### .purple[**`unbreak_rows()`**]

???

Let's fix some broken values. For this we'll use functions from the unheadr package. First is the unbreak rows function.
This function will merge two rows by matching the leading half of the broken value with a regular expression.

Verb for merging rows from 📦 .rrured.b[`unheadr`]

Match the leading half of the broken value with .orange.b[regular expression]

```r
unbreak_rows(df,
  regex,
  ogcol,
  sep = " "
)
```

???

This function takes a data object and three important arguments. A regular expression to match the leading parts of a broken value, the column we'll be working on, and the separator that we want between the two parts of the value when we paste them together.

---

## Leading row

```r
olympics %>% unbreak_rows(
  regex = "^weight", # string starts with 'weight'
  ogcol = `Wrestling.gold.medal.(men.middleweight)`,
  sep = " ")
```

```
## # A tibble: 9 x 4
##   Edition        Country    `Soccer.gold.medal.(… `Wrestling.gold.medal.(men.mi…
##   <chr>          <chr>      <chr>                 <chr>                         
## 1 ""             ""         ""                    weight class limit: 82 kg     
## 2 "Los Angeles … "USA"      "France"              USA                           
## 3 "Barcelona"    "Spain"    "Spain"               USA                           
## 4  <NA>           <NA>       <NA>                 <NA>                          
## 5 "Atlanta 1996" "USA"      "Nigeria"             Russia                        
## 6 ""             ""         ""                    weight class limit: 85 kg     
## 7 "Sydney 2000"  "Australi… "Cameroon"            Russia                        
## 8 "London"       "UK"       "Mexico"              Azerbaijan                    
## 9  <NA>           <NA>       <NA>                 <NA>
```

???

This is how we fix the broken values in the wrestling gold medal column. There was information about changes to the weight class intermingled in this column. To match the rows that have the first half of a broken value, we can use a regular expression for strings that start with the literal string weight. A space can be a good separator here. When this runs, notice that the broken values are now in a single row and the table is more compact.

---

## .purple[**`unbreak_vals()`**]

???

To fix broken values by matching the trailing row, we use unbreak vals from headr.

Match the trailing half of the broken value with a .orange.b[regular expression]
]

```r
unbreak_vals(df,
  regex,
  ogcol,
  newcol,
  sep = " ")
```

???

This function takes a data object and four arguments. A regular expression to match the trailing parts of a broken value, the column we'll be working on, and the name for a new column to create with the unified values. Finally, the separator that we want between the two parts of the value when we paste them together.

---

## Trailing row

```r
olympics %>%
  unbreak_vals(
    regex = "^\\d", # numbers at start of string
    ogcol = Edition,
    newcol = Edition_ub,
    sep = " ")
```

```
## # A tibble: 9 x 4
##   Edition_ub     Country   `Soccer.gold.medal.(… `Wrestling.gold.medal.(men.mid…
##   <chr>          <chr>     <chr>                 <chr>                          
## 1 <NA>           <NA>      <NA>                  weight class limit:            
## 2 <NA>           <NA>      <NA>                  82 kg                          
## 3 Los Angeles 1… USA       France                USA                            
## 4 Barcelona 1992 Spain     Spain                 USA                            
## 5 Atlanta 1996   USA       Nigeria               Russia                         
## 6 <NA>           <NA>      <NA>                  weight class limit:            
## 7 <NA>           <NA>      <NA>                  85 kg                          
## 8 Sydney 2000    Australia Cameroon              Russia                         
## 9 London 2012    UK        Mexico                Azerbaijan
```

???

This is how we fix the broken values in the edition variable. To match the rows that have the second half of a broken value, we can use a regular expression for strings that start with numbers. We specify that we want to work with the edition column, and I set the new unified values to be in a column called edition underscore ub, for unbroken. We'll also use a space as a separator. When this runs, the unified values are now in a single row, in the new column, which by default is placed at the beginning of the data.

---

## Unbreak sequentially

???

We can chain these two operations together with pipes. With the same arguments as before, we pass the output from one function to the other one.

```r
olympics %>%
  unbreak_vals(
    regex = "^\\d", # numbers at start of string
    ogcol = Edition,
    newcol = Edition_ub
  ) %>% 
    unbreak_rows(
    regex = "^weight",
    ogcol = `Wrestling.gold.medal.(men.middleweight)`)
```

---

## Unbreak sequentially

```
## # A tibble: 7 x 4
##   Edition_ub      Country   `Soccer.gold.medal.(… `Wrestling.gold.medal.(men.mi…
##   <chr>           <chr>     <chr>                 <chr>                         
## 1 ""              ""        ""                    weight class limit: 82 kg     
## 2 "Los Angeles 1… "USA"     "France"              USA                           
## 3 "Barcelona 199… "Spain"   "Spain"               USA                           
## 4 "Atlanta 1996"  "USA"     "Nigeria"             Russia                        
## 5 ""              ""        ""                    weight class limit: 85 kg     
## 6 "Sydney 2000"   "Austral… "Cameroon"            Russia                        
## 7 "London 2012"   "UK"      "Mexico"              Azerbaijan
```

???

The result looks much better now, and we already know how to fix the two remaining issues: the messy names, and the data embedded as weird subheaders.

---
class: my-turn

# My turn

---

---

class: center, middle, dk-section-title
background-image:url("images/david-hertle-8HAhmMk9HJI-unsplash.jpg")
background-size: cover

# Empty Rows and Columns

???

In general, data organization good practices suggest not leaving empty cells, and also being aware of how we encode missing data. Empty rows and columns are an extension of this. Let's see how we can sort them out.

---

### Problems with empty rows and columns

???
Empty here refers to rows or columns in our data for which all values are missing.

--
.large[
- Common when importing files
]

???

It's quite common to find empty rows or columns when we import files, and the reasons for them being prsent can vary.

.large[
- Problematic when referring to variables by position or ranges of consecutive variables  (`:`)  
]

???

Having empty columns can affect our work because they might interfere with how we select variables by position or when we work with ranges of contiguous variables. Also, when we perform operations across observations, empty columns might lead to unexpected errors.

???
Empty rows are also inconvenient if we need to fill adjacent values of perform other row-wise operations. Fortunately, we have options for cleaning data with empty rows or columns.

---

# What can we do?

???

To deal with empty rows or columns, we can use a couple of familiar packages to first identify them, then discard them, because we don't need empty rows or columns in our data.

???

We can identify empty rows or columns with functions from dplyr,

???

and remove them with the remove empty function from janitor.

---

.panelset[
.panel[.panel-name[universities]
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#gwodbannqj .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #f4f4f9;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#gwodbannqj .gt_heading {
  background-color: #f4f4f9;
  text-align: center;
  border-bottom-color: #f4f4f9;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#gwodbannqj .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #f4f4f9;
  border-bottom-width: 0;
}

#gwodbannqj .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #f4f4f9;
  border-top-width: 0;
}

#gwodbannqj .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#gwodbannqj .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#gwodbannqj .gt_col_heading {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#gwodbannqj .gt_column_spanner_outer {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#gwodbannqj .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#gwodbannqj .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#gwodbannqj .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#gwodbannqj .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#gwodbannqj .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#gwodbannqj .gt_from_md > :first-child {
  margin-top: 0;
}

#gwodbannqj .gt_from_md > :last-child {
  margin-bottom: 0;
}

#gwodbannqj .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#gwodbannqj .gt_stub {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#gwodbannqj .gt_summary_row {
  color: #333333;
  background-color: #f4f4f9;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#gwodbannqj .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#gwodbannqj .gt_grand_summary_row {
  color: #333333;
  background-color: #f4f4f9;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#gwodbannqj .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#gwodbannqj .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#gwodbannqj .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#gwodbannqj .gt_footnotes {
  color: #333333;
  background-color: #f4f4f9;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#gwodbannqj .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#gwodbannqj .gt_sourcenotes {
  color: #333333;
  background-color: #f4f4f9;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#gwodbannqj .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#gwodbannqj .gt_left {
  text-align: left;
}

#gwodbannqj .gt_center {
  text-align: center;
}

#gwodbannqj .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#gwodbannqj .gt_font_normal {
  font-weight: normal;
}

#gwodbannqj .gt_font_bold {
  font-weight: bold;
}

#gwodbannqj .gt_font_italic {
  font-style: italic;
}

#gwodbannqj .gt_super {
  font-size: 65%;
}

#gwodbannqj .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="gwodbannqj" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 14px;">ID</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 14px;">Institution</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1" style="font-size: 14px;">year</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 14px;">ZIP.code</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 14px;">Highest.degree.offered</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1" style="font-size: 14px;">County.name</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 14px;">Religious.affiliation</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_right" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">100663</td>
<td class="gt_row gt_left" style="font-size: 14px;">University of Alabama at Birmingham</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">35294-0110</td>
<td class="gt_row gt_left" style="font-size: 14px;">Doctor's degree</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">Not applicable</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">100690</td>
<td class="gt_row gt_left" style="font-size: 14px;">Amridge University</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">36117-3553</td>
<td class="gt_row gt_left" style="font-size: 14px;">Doctor's degree</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">Churches of Christ</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">100706</td>
<td class="gt_row gt_left" style="font-size: 14px;">University of Alabama in Huntsville</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">35899</td>
<td class="gt_row gt_left" style="font-size: 14px;">Doctor's degree</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">Not applicable</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">100751</td>
<td class="gt_row gt_left" style="font-size: 14px;">The University of Alabama</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">35487-0166</td>
<td class="gt_row gt_left" style="font-size: 14px;">Doctor's degree</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">Not applicable</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">101541</td>
<td class="gt_row gt_left" style="font-size: 14px;">Judson College</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">36756</td>
<td class="gt_row gt_left" style="font-size: 14px;">Bachelor's degree</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">Baptist</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">101587</td>
<td class="gt_row gt_left" style="font-size: 14px;">University of West Alabama</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">35470</td>
<td class="gt_row gt_left" style="font-size: 14px;">Master's degree</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">Not applicable</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">NA</td></tr>
    <tr><td class="gt_row gt_right" style="font-size: 14px;">101693</td>
<td class="gt_row gt_left" style="font-size: 14px;">University of Mobile</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">36613-2842</td>
<td class="gt_row gt_left" style="font-size: 14px;">Master's degree</td>
<td class="gt_row gt_center" style="font-size: 14px;">NA</td>
<td class="gt_row gt_left" style="font-size: 14px;">Southern Baptist</td></tr>
  </tbody>
  
  
</table></div>
]

```r
universities <-
  tibble::tribble(
    ~ID, ~Institution, ~year, ~ZIP.code, ~Highest.degree.offered, ~County.name, ~Religious.affiliation,
    NA, NA, NA, NA, NA, NA, NA,
    100663L, "University of Alabama at Birmingham", NA, "35294-0110", "Doctor's degree", NA, "Not applicable",
    100690L, "Amridge University", NA, "36117-3553", "Doctor's degree", NA, "Churches of Christ",
    100706L, "University of Alabama in Huntsville", NA, "35899", "Doctor's degree", NA, "Not applicable",
    NA, NA, NA, NA, NA, NA, NA,
    100751L, "The University of Alabama", NA, "35487-0166", "Doctor's degree", NA, "Not applicable",
    NA, NA, NA, NA, NA, NA, NA,
    NA, NA, NA, NA, NA, NA, NA,
    101541L, "Judson College", NA, "36756", "Bachelor's degree", NA, "Baptist",
    101587L, "University of West Alabama", NA, "35470", "Master's degree", NA, "Not applicable",
    NA, NA, NA, NA, NA, NA, NA,
    101693L, "University of Mobile", NA, "36613-2842", "Master's degree", NA, "Southern Baptist"
  )
```
]
]
.small[source: US Integrated Postsecondary Education Data System (IPEDS)]

???

Let's look at these example data. This object, called universities, includes details about some universities in the united states. There are a lot of NAs in these data, and if we look more closely, some rows and columns are empty altogether.

---

## Identify empty rows with 📦 .rrured.b[`dplyr`] (and 📦 .rrured.b[`tibble`])

???

In smaller datasets we can simply look at them and see which rows or columns are empty. With larger volumes of data this is not possible, so we can existing functions to identify them.

Let's do rows first.
For this, we'll use the filter function from dplyr, which we used before to subset data and keep only rows that meet a condition.

```r
universities %>% filter(across(everything(), is.na))
```

```
## # A tibble: 5 x 7
##      ID Institution year  ZIP.code Highest.degree.… County.name Religious.affil…
##   <int> <chr>       <lgl> <chr>    <chr>            <lgl>       <chr>           
## 1    NA <NA>        NA    <NA>     <NA>             NA          <NA>            
## 2    NA <NA>        NA    <NA>     <NA>             NA          <NA>            
## 3    NA <NA>        NA    <NA>     <NA>             NA          <NA>            
## 4    NA <NA>        NA    <NA>     <NA>             NA          <NA>            
## 5    NA <NA>        NA    <NA>     <NA>             NA          <NA>
```

???
To use the filter function on many columns at once, we use the new across syntax, and the helper function everything to work across all the variables. The is.na function, applied to each rows, will tell us if ALL the values in the row are NA or not. If they are, then we keep them.

The result from this was a tibble with five rows, which are all empty.

This tells us that there were empty rows in the data, and how many, but we can also use a function from the tibble package to find out which ones.

---

### Add unique identifier first

```r
universities %>%
* rowid_to_column() %>%
  filter(across(-rowid, is.na))
```

```
## # A tibble: 5 x 8
##   rowid    ID Institution year  ZIP.code Highest.degree.offered County.name
##   <int> <int> <chr>       <lgl> <chr>    <chr>                  <lgl>      
## 1     1    NA <NA>        NA    <NA>     <NA>                   NA         
## 2     5    NA <NA>        NA    <NA>     <NA>                   NA         
## 3     7    NA <NA>        NA    <NA>     <NA>                   NA         
## 4     8    NA <NA>        NA    <NA>     <NA>                   NA         
## 5    11    NA <NA>        NA    <NA>     <NA>                   NA         
## # … with 1 more variable: Religious.affiliation <chr>
```

???

We can add a unique identifier to each row before subsetting to keep only the empty rows. The row id to column function from the package tibble takes the row numbering inherent to the data and places it as a new column. 
Then we can filter for rows which are empty in all variables except in the new one we just created. By default, its name is rowid.

Now we know the position of the five empty rows, but how do we remove them.

---

### Negate the selection .b.purple[(!)] and discard empty rows

???

To remove the empty rows, we follow the same process as before, but instead of keeping the empty rows, we discard them by getting the complement of our selection with an exclamation mark.

```r
universities %>%
  filter(!across(everything(), is.na))
```

```
## # A tibble: 7 x 7
##       ID Institution             year  ZIP.code  Highest.degree.off… County.name
##    <int> <chr>                   <lgl> <chr>     <chr>               <lgl>      
## 1 100663 University of Alabama … NA    35294-01… Doctor's degree     NA         
## 2 100690 Amridge University      NA    36117-35… Doctor's degree     NA         
## 3 100706 University of Alabama … NA    35899     Doctor's degree     NA         
## 4 100751 The University of Alab… NA    35487-01… Doctor's degree     NA         
## 5 101541 Judson College          NA    36756     Bachelor's degree   NA         
## 6 101587 University of West Ala… NA    35470     Master's degree     NA         
## 7 101693 University of Mobile    NA    36613-28… Master's degree     NA         
## # … with 1 more variable: Religious.affiliation <chr>
```

???

This will keep only the rows that aren't empty.

---

???

I just showed a way to remove empty rows with dplyr, but there is an easier alternative, using the remove empty function from the janitor package.

> `quiet = FALSE` prints a statement about the rows that were removed

> `'which'` specifies rows or columns

```r
universities %>%
  remove_empty(
    which = "rows",
    quiet = FALSE)
```

???

We simply specify if we want to remove row or columns, and there is an argument called quiet that controls if a little message with details prints to the console.

---

```r
universities %>%
  remove_empty(
    which = "rows",
    quiet = FALSE)
```

```
## Removing 5 empty rows of 12 rows total (41.7%).
```

???

Using remove empty and specifying rows in the which argument, we get the same result as before, and because we set the quiet argument to FALSE, a small message tells us that five empty rows were removed out of twelve in total. This looks much better.

---

## Empty columns

???

Now let's deal with empty columns in our data.

```r
universities %>%
  select(where(all_na))
```

**`where`** helps select variables for which a function returns `TRUE`

**`all_na`** checks if all values in a vector are `NA`

???

Before we remove them, we can identify then with functions from dplyr and naniar. We use select from dplyr to subset columns, and the where helper to keep only the variables that meet a condition. The condition here is if the entire column is all NA. We specificy this with all na from the package naniar.

---

```r
universities %>%
  select(where(all_na))
```

```
## # A tibble: 12 x 2
##    year  County.name
##    <lgl> <lgl>      
##  1 NA    NA         
##  2 NA    NA         
##  3 NA    NA         
##  4 NA    NA         
##  5 NA    NA         
##  6 NA    NA         
##  7 NA    NA         
##  8 NA    NA         
##  9 NA    NA         
## 10 NA    NA         
## 11 NA    NA         
## 12 NA    NA
```

???
This gives a subset of the data, but only with the empty columns. Because columns have names, we know right away which ones we can remove.

---

### Identify and get names

```r
universities %>%
  select(where(all_na)) %>%
  names()
```

```
## [1] "year"        "County.name"
```

???

If we want to skip printing the object and just get the names, we can, and the names function at the end of the sequence will gives us a vector with the names of the empty columns.

---

### Negate selection, get names

???

Like we did with rows, we get the complement of our selection and this will remove the empty ones.

```r
universities %>%
  select(!where(all_na)) %>%
  names()
```

```
## [1] "ID"                     "Institution"            "ZIP.code"              
## [4] "Highest.degree.offered" "Religious.affiliation"
```

???

With this code, we know which variables aren't empty.

---

### Discard empty columns with .b.purple[`remove_empty()`]

???

We can also use janitor and the remove empty function for this.

```r
universities %>%
    remove_empty(which = "cols", quiet = FALSE)
```

> `quiet = FALSE` prints a statement about the rows that were removed

> `'which'` specifies rows or columns

???

The syntax is simple, we specify cols in the which argument and run the code.

---

#### Discard empty columns with .b.purple[`remove_empty()`]

```r
universities %>%
    remove_empty(which = "cols", quiet = FALSE)
```

```
## Removing 2 empty columns of 7 columns total (Removed: year, County.name).
```

```
## # A tibble: 12 x 5
##        ID Institution            ZIP.code  Highest.degree.of… Religious.affilia…
##     <int> <chr>                  <chr>     <chr>              <chr>             
##  1     NA <NA>                   <NA>      <NA>               <NA>              
##  2 100663 University of Alabama… 35294-01… Doctor's degree    Not applicable    
##  3 100690 Amridge University     36117-35… Doctor's degree    Churches of Christ
##  4 100706 University of Alabama… 35899     Doctor's degree    Not applicable    
##  5     NA <NA>                   <NA>      <NA>               <NA>              
##  6 100751 The University of Ala… 35487-01… Doctor's degree    Not applicable    
##  7     NA <NA>                   <NA>      <NA>               <NA>              
##  8     NA <NA>                   <NA>      <NA>               <NA>              
##  9 101541 Judson College         36756     Bachelor's degree  Baptist           
## 10 101587 University of West Al… 35470     Master's degree    Not applicable    
## 11     NA <NA>                   <NA>      <NA>               <NA>              
## 12 101693 University of Mobile   36613-28… Master's degree    Southern Baptist
```

???

Quiet is set to false, so we'll get a message telling us how many and which ones were removed.

---
### Discard empty rows and columns simultaneously with .b.purple[`remove_empty()`]

```r
universities %>%
    remove_empty(which = c("cols","rows"), quiet = TRUE)
```

```
## # A tibble: 7 x 5
##       ID Institution             ZIP.code  Highest.degree.of… Religious.affilia…
##    <int> <chr>                   <chr>     <chr>              <chr>             
## 1 100663 University of Alabama … 35294-01… Doctor's degree    Not applicable    
## 2 100690 Amridge University      36117-35… Doctor's degree    Churches of Christ
## 3 100706 University of Alabama … 35899     Doctor's degree    Not applicable    
## 4 100751 The University of Alab… 35487-01… Doctor's degree    Not applicable    
## 5 101541 Judson College          36756     Bachelor's degree  Baptist           
## 6 101587 University of West Ala… 35470     Master's degree    Not applicable    
## 7 101693 University of Mobile    36613-28… Master's degree    Southern Baptist
```

???

If we want to remove empty rows and columns all at once, we can pass a character vector with cols and rows to remove empty. For this one I set quiet to TRUE, so there's no message printed to the console.

---
class: my-turn

# My turn

- Import the Marine Protected Areas dataset (MPAS-mine.csv)

- Identify the empty rows and columns

- Remove the empty rows and columns

---
class: inverse

# Your turn

- Import the Marine Protected Areas dataset (MPAS-your.csv)

- Identify the empty rows and columns

- Remove the empty rows and columns

---

class: center, middle, dk-section-title
background-image:url("images/markus-spiske-4jiR57y3jgY-unsplash.jpg")
background-size: cover
# Parsing Numbers

???

In this lesson, we'll learn how to make numeric values in our data usable, because they often aren't.

We can start by going over some basic R.

---
class: middle

???

In R, vectors must have all their values of the same mode, which can be character, numeric, or logical.

.large[
- If there is a character string is present in a vector, everything else in the vector will be converted to character strings
]

???
Because everything in a vector must be of the same type, if we have character strings in a vector then everything will be treated as type character, including numbers.

---

### We cannot perform arithmetic operations on:

???

If this happens, then we're blocked from going further, because we can't do arithmetic operations on

- Numbers stored as text

???

Numbers stored as text,

- Strings with non-digits

???

or strings with non-digits.

```r
test_scores <- c(8.8, 9, 10, 7.2, 8.4)
class(test_scores)
```

```
## [1] "numeric"
```

```r
class(c(test_scores, "a"))
```

```
## [1] "character"
```

???

This is what I mean. First we have a vector of five numbers, called test scores. This is an object of type numeric, but it becomes a character vector as soon as we mix in a string. In this case, the letter a.

---

## Numbers stored as text

???

In R, it is possible to have columns in our data that are of type character, even when they contain only numbers. This is like using cell format in a spreadsheet. If this is the case, the conversion to numeric is simple and we have two options.

1. .b.purple[`type_convert()`] the whole object (📦 .rrured.b[`readr`])

???

First, we can use the type convert function from readr to take an object, look at all the content in each column, and guess the type for each one. This works great for columns with all numbers.

2. Coerce one or more variables to `numeric` with 📦 .rrured.b[`dplyr`]

???

Another option is to coerce one or more variables to type numeric using dplyr. In this case we're transforming the data and applying a coercion to those columns that we believe to be numbers stored as text.

---

## Strings with non-digits

???

We cannot perform numeric operations on numbers that have non-digits. To get rid of non digits, we have two options as well.

1. Parse with 📦 .rrured.b[`readr`]

???

We can parse our variables with functions from readr,

--
2. Clean with .orange.b[regular expressions]

???

Or we can clean up the values with regular expressions.

---

.panelset[
.panel[.panel-name[Burger Prices]
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#rrtlykjoyj .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #f4f4f9;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#rrtlykjoyj .gt_heading {
  background-color: #f4f4f9;
  text-align: center;
  border-bottom-color: #f4f4f9;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#rrtlykjoyj .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #f4f4f9;
  border-bottom-width: 0;
}

#rrtlykjoyj .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 4px;
  border-top-color: #f4f4f9;
  border-top-width: 0;
}

#rrtlykjoyj .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#rrtlykjoyj .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#rrtlykjoyj .gt_col_heading {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#rrtlykjoyj .gt_column_spanner_outer {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#rrtlykjoyj .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#rrtlykjoyj .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#rrtlykjoyj .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#rrtlykjoyj .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#rrtlykjoyj .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#rrtlykjoyj .gt_from_md > :first-child {
  margin-top: 0;
}

#rrtlykjoyj .gt_from_md > :last-child {
  margin-bottom: 0;
}

#rrtlykjoyj .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#rrtlykjoyj .gt_stub {
  color: #333333;
  background-color: #f4f4f9;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#rrtlykjoyj .gt_summary_row {
  color: #333333;
  background-color: #f4f4f9;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#rrtlykjoyj .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#rrtlykjoyj .gt_grand_summary_row {
  color: #333333;
  background-color: #f4f4f9;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#rrtlykjoyj .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#rrtlykjoyj .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#rrtlykjoyj .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#rrtlykjoyj .gt_footnotes {
  color: #333333;
  background-color: #f4f4f9;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#rrtlykjoyj .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#rrtlykjoyj .gt_sourcenotes {
  color: #333333;
  background-color: #f4f4f9;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#rrtlykjoyj .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#rrtlykjoyj .gt_left {
  text-align: left;
}

#rrtlykjoyj .gt_center {
  text-align: center;
}

#rrtlykjoyj .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#rrtlykjoyj .gt_font_normal {
  font-weight: normal;
}

#rrtlykjoyj .gt_font_bold {
  font-weight: bold;
}

#rrtlykjoyj .gt_font_italic {
  font-style: italic;
}

#rrtlykjoyj .gt_super {
  font-size: 65%;
}

#rrtlykjoyj .gt_footnote_marks {
  font-style: italic;
  font-size: 65%;
}
</style>
<div id="rrtlykjoyj" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;"><table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 24px;">Rank</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 24px;">Country</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 24px;">Price</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 24px;">Price.ARS</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 24px;">Region</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left" style="font-size: 24px;">1</td>
<td class="gt_row gt_left" style="font-size: 24px;">Switzerland</td>
<td class="gt_row gt_left" style="font-size: 24px;">6.8</td>
<td class="gt_row gt_left" style="font-size: 24px;">$544.50</td>
<td class="gt_row gt_left" style="font-size: 24px;">Europe</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 24px;">2</td>
<td class="gt_row gt_left" style="font-size: 24px;">Norway</td>
<td class="gt_row gt_left" style="font-size: 24px;">6.2</td>
<td class="gt_row gt_left" style="font-size: 24px;">496</td>
<td class="gt_row gt_left" style="font-size: 24px;">Europe</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 24px;">3</td>
<td class="gt_row gt_left" style="font-size: 24px;">Sweden</td>
<td class="gt_row gt_left" style="font-size: 24px;">6.1</td>
<td class="gt_row gt_left" style="font-size: 24px;">488</td>
<td class="gt_row gt_left" style="font-size: 24px;">Europe</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 24px;">4</td>
<td class="gt_row gt_left" style="font-size: 24px;">Finland</td>
<td class="gt_row gt_left" style="font-size: 24px;">5.6</td>
<td class="gt_row gt_left" style="font-size: 24px;">448</td>
<td class="gt_row gt_left" style="font-size: 24px;">Europe</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 24px;">5</td>
<td class="gt_row gt_left" style="font-size: 24px;">United States</td>
<td class="gt_row gt_left" style="font-size: 24px;">5.3[1]</td>
<td class="gt_row gt_left" style="font-size: 24px;">$424.50</td>
<td class="gt_row gt_left" style="font-size: 24px;">America</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 24px;">8</td>
<td class="gt_row gt_left" style="font-size: 24px;">Italy</td>
<td class="gt_row gt_left" style="font-size: 24px;">5.1</td>
<td class="gt_row gt_left" style="font-size: 24px;">408</td>
<td class="gt_row gt_left" style="font-size: 24px;">Europe</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 24px;">7</td>
<td class="gt_row gt_left" style="font-size: 24px;">France</td>
<td class="gt_row gt_left" style="font-size: 24px;">5.1</td>
<td class="gt_row gt_left" style="font-size: 24px;">408,50</td>
<td class="gt_row gt_left" style="font-size: 24px;">Europe</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 24px;">6</td>
<td class="gt_row gt_left" style="font-size: 24px;">Canada</td>
<td class="gt_row gt_left" style="font-size: 24px;">5.3</td>
<td class="gt_row gt_left" style="font-size: 24px;">424</td>
<td class="gt_row gt_left" style="font-size: 24px;">America</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 24px;">9</td>
<td class="gt_row gt_left" style="font-size: 24px;">Brazil</td>
<td class="gt_row gt_left" style="font-size: 24px;">5.1*</td>
<td class="gt_row gt_left" style="font-size: 24px;">408</td>
<td class="gt_row gt_left" style="font-size: 24px;">America</td></tr>
  </tbody>
  
  
</table></div>
]

```r
burger_prices <-
  tibble::tribble(
    ~Rank, ~Country, ~Price, ~Price.ARS, ~Region,
    "1", "Switzerland", "6.8", "$544.50", "Europe",
    "2", "Norway", "6.2", "496", "Europe",
    "3", "Sweden", "6.1", "488", "Europe",
    "4", "Finland", "5.6", "448", "Europe",
    "5", "United States", "5.3[1]", "$424.50", "America",
    "8", "Italy", "5.1", "408", "Europe",
    "7", "France", "5.1", "408,50", "Europe",
    "6", "Canada", "5.3", "424", "America",
    "9", "Brazil", "5.1*", "408", "America"
  )
```
]
]

???

Let's see the example data for this lesson. This object called burger prices has prices for the same hamburger in different countries, shown in two different currencies.

---

```r
burger_prices
```

```
## # A tibble: 9 x 5
##   Rank  Country       Price  Price.ARS Region 
##   <chr> <chr>         <chr>  <chr>     <chr>  
## 1 1     Switzerland   6.8    $544.50   Europe 
## 2 2     Norway        6.2    496       Europe 
## 3 3     Sweden        6.1    488       Europe 
## 4 4     Finland       5.6    448       Europe 
## 5 5     United States 5.3[1] $424.50   America
## 6 8     Italy         5.1    408       Europe 
## 7 7     France        5.1    408,50    Europe 
## 8 6     Canada        5.3    424       America
## 9 9     Brazil        5.1*   408       America
```

???

Printed in the console, the data looks like this. The printing methods for tibble objects tell us the data type for each column, and in these data all the columns are of type character, including the ones that contain numbers and including the first column - Rank, which just has numbers from one to nine.

---

## Numbers stored as text

???

The Rank column in our data has numbers stored as text. There is no good reason for this variable to be of type character, so we can make it numeric.

Parse all columns

```r
burger_prices %>%
  type_convert()
```

```
## # A tibble: 9 x 5
##    Rank Country       Price  Price.ARS Region 
##   <dbl> <chr>         <chr>  <chr>     <chr>  
## 1     1 Switzerland   6.8    $544.50   Europe 
## 2     2 Norway        6.2    496       Europe 
## 3     3 Sweden        6.1    488       Europe 
## 4     4 Finland       5.6    448       Europe 
## 5     5 United States 5.3[1] $424.50   America
## 6     8 Italy         5.1    408       Europe 
## 7     7 France        5.1    408,50    Europe 
## 8     6 Canada        5.3    424       America
## 9     9 Brazil        5.1*   408       America
```

???

Our first approach should be to just parse all the columns in the object with the type convert function from readr. When we do this, we can see that the Rank column is now a variable of type double, which is a class of of values within numeric.

---

## Numbers stored as text

???

For a finer control of our data, we could just coerce the problematic variable to numeric. R has a function for this called as.numeric.

Coerce variables to `numeric`

```r
burger_prices %>%
  mutate(Rank = as.numeric(Rank))
```

???

To change a single variable, we can use mutate from dplyr to overwrite Rank, in this case, with the values coerced to numeric using as.numeric. The output is the same as before.

Ok, so the rank column is numeric now, but we still want to fix the values with the actual prices. Looking at these column, we see that one of them has annotations and special characters, the other one seems to have inconsistent decimal marks, and only a few of the prices has dollar signs to indicate currency.

---

## Strings with non-digits

???

The two columns with prices are good examples of numbers stored as strings with non-digits. We can't work with these prices until we parse the numbers in these variables into something usable.

Parse with .b.purple[`parse_number`] from 📦 .rrured.b[`readr`] (⚠ watch out for inconsistent decimals)

???

We can first use the parse number function from readr and see how we do. Parse number drops any non-numeric characters from a string, trying to respect how commas or dots are used to separate decimals or group digits for larger numbers.

```r
burger_prices %>%
  mutate(across(c(Rank, Price, Price.ARS), parse_number))
```

```
## # A tibble: 9 x 5
##    Rank Country       Price Price.ARS Region 
##   <dbl> <chr>         <dbl>     <dbl> <chr>  
## 1     1 Switzerland   6.8       544.5 Europe 
## 2     2 Norway        6.2       496   Europe 
## 3     3 Sweden        6.1       488   Europe 
## 4     4 Finland       5.6       448   Europe 
## 5     5 United States 5.3       424.5 America
## 6     8 Italy         5.100     408   Europe 
## 7     7 France        5.100   40850   Europe 
## 8     6 Canada        5.3       424   America
## 9     9 Brazil        5.100     408   America
```

???
In the code, we use mutate and the across syntax from dpyr to modify many columns at once, we specify Rank and the two variables with the price, and that we want to apply the parse number function to each one.

The result looks good for the Rank and Price columns. The extra comments or annotations in the Price column were dropped and now this column appears as a variable of type double.

The price dot ARS column is also of type double now. The dollar signs were stripped, but because of an inconsistent decimal point, the values are off now. We need to be careful with this.

---

## Strings with non-digits

???

Let's try again, but using string manipulation with an approach suited for these messy data.

```r
burger_prices %>%
  mutate(across(c(Rank, Price, Price.ARS), str_remove_all, "\\[.+\\]|\$.+\$")) %>%
  mutate(across(c(Rank, Price, Price.ARS), str_remove_all, "[^0-9.,]")) %>%
  mutate(Price.ARS = str_replace(Price.ARS, ",", ".")) %>%
  type_convert()
```

???

In the first step, we use mutate, the across function, combined the str remove all function from stringr and a regular expression. This will match any notes or additional details in brackets that may be present in our target columns.

The second step is similar, but the regular expression is a negated character set for matching anything that is not a number, a comma, or a dot. We don't want any of these in our numeric data.

In the third step we replace commas with dots only in Price dot ARS, and finally we type convert everything.

---

```
## # A tibble: 9 x 5
##    Rank Country       Price Price.ARS Region 
##   <dbl> <chr>         <dbl>     <dbl> <chr>  
## 1     1 Switzerland   6.8       544.5 Europe 
## 2     2 Norway        6.2       496   Europe 
## 3     3 Sweden        6.1       488   Europe 
## 4     4 Finland       5.6       448   Europe 
## 5     5 United States 5.3       424.5 America
## 6     8 Italy         5.100     408   Europe 
## 7     7 France        5.100     408.5 Europe 
## 8     6 Canada        5.3       424   America
## 9     9 Brazil        5.100     408   America
```

???

The resulting object has usable numeric variables. We can now work with these prices, for example by converting them to other currencies through multiplication, or finding the mean price by region.

---
class: my-turn
# My turn

--
.large[
- Import the Marine Protected Areas dataset (MPAS-mine.csv)
]

---
class: inverse
# Your turn

.large[
- Make the columns that hold the MPA extent into usable numeric variables (Watch out for the decimals!)]

---

class: center, middle, dk-section-title
background-image:url("images/ryan-quintal-US9Tc9pKNBU-unsplash.jpg")
background-size: cover

# Putting Everything Together

---

## Demonstration