Webscraping

]

# Week 12: .fancy[Webscraping]

### <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M243.4 2.6l-224 96c-14 6-21.8 21-18.7 35.8S16.8 160 32 160v8c0 13.3 10.7 24 24 24H456c13.3 0 24-10.7 24-24v-8c15.2 0 28.3-10.7 31.3-25.6s-4.8-29.9-18.7-35.8l-224-96c-8.1-3.4-17.2-3.4-25.2 0zM128 224H64V420.3c-.6 .3-1.2 .7-1.8 1.1l-48 32c-11.7 7.8-17 22.4-12.9 35.9S17.9 512 32 512H480c14.1 0 26.5-9.2 30.6-22.7s-1.1-28.1-12.9-35.9l-48-32c-.6-.4-1.2-.7-1.8-1.1V224H384V416H344V224H280V416H232V224H168V416H128V224zm128-96c-17.7 0-32-14.3-32-32s14.3-32 32-32s32 14.3 32 32s-14.3 32-32 32z"/></svg> EMSE 4571: Intro to Programming for Analytics
### <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M272 304h-96C78.8 304 0 382.8 0 480c0 17.67 14.33 32 32 32h384c17.67 0 32-14.33 32-32C448 382.8 369.2 304 272 304zM48.99 464C56.89 400.9 110.8 352 176 352h96c65.16 0 119.1 48.95 127 112H48.99zM224 256c70.69 0 128-57.31 128-128c0-70.69-57.31-128-128-128S96 57.31 96 128C96 198.7 153.3 256 224 256zM224 48c44.11 0 80 35.89 80 80c0 44.11-35.89 80-80 80S144 172.1 144 128C144 83.89 179.9 48 224 48z"/></svg> John Paul Helveston
### <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:white;overflow:visible;position:relative;"><path d="M152 64H296V24C296 10.75 306.7 0 320 0C333.3 0 344 10.75 344 24V64H384C419.3 64 448 92.65 448 128V448C448 483.3 419.3 512 384 512H64C28.65 512 0 483.3 0 448V128C0 92.65 28.65 64 64 64H104V24C104 10.75 114.7 0 128 0C141.3 0 152 10.75 152 24V64zM48 448C48 456.8 55.16 464 64 464H384C392.8 464 400 456.8 400 448V192H48V448z"/></svg> April 20, 2023

]

---

# Week 12: .fancy[Webscraping]

### 1. Scraping static pages
### 2. Scraping multiple pages

### BREAK

### 3. Using APIs

---

#### Some disclaimers ([here](https://r4ds.hadley.nz/webscraping.html#scraping-ethics-and-legalities) for more details)

You're probably okay if the data is:

- Public
- Non-personal
- Factual

Otherwise, consult a lawyer and / or maybe don't scrape it.

#### Terms of service

Generally are not upheld, unless you **need an account to access the data**.

#### Copyright

Data is not copyright protected (in the US). But works are. Be careful.

---

# Week 12: .fancy[Webscraping]

### 1. HTML basics
### 1. Scraping static pages
### 2. Scraping multiple pages

### BREAK

### 3. Using APIs

---

## **H**yper**T**ext **M**arkup **L**anguage

```html
<html>
<head>
  <title>Page title</title>
</head>
<body>
  <h1 id='first'>A heading</h1>
  <p>Some text &amp; <b>some bold text.</b></p>
  <img src='myimg.png' width='100' height='100'>
</body>
```

HTML has a hierarchical structure formed by:

- Start and end **"tags"** (e.g. `<tag>` and `</tag>`)
- Optional attributes (e.g. `id='first'`)
- Contents (everything in between the start and end tag).

---

## Common tags

- `<h1>` = Header level 1
- `<a>` = [Url]() link
- `<b>` = **Bold** text 
- `<i>` = _Italic_ text
- `<p>` = Paragraph
- `<li>` = List item

]

## Attributes

- `id`: Element identifier, e.g.<br>`<h1 id='first'>A heading</h1>`
- `class`: Styling class, e.g.<br>`<h1 class='header'>A heading</h1>`

]

---

# Quick example

- Go [here](https://rvest.tidyverse.org/articles/starwars.html)
- Right-click, select<br>"View Page Source"

]

https://rvest.tidyverse.org/articles/starwars.html
<center>
<img src="images/view-source.png" width=100%>
</center>

]

---

## **Strategy**: Use tags and classes to parse html

`source_code`

```html
<html>
<head>
  <title>Page title</title>
</head>
<body>
* <h1 id='first'>A heading</h1>
  <p>Some text &amp; <b>some bold text.</b></p>
  <img src='myimg.png' width='100' height='100'>
</body>
```

]

Use `{rvest}` package to parse html

```r
library(rvest)

html <- read_html(source_code)

html %>% 
* html_elements("h1")
```

```
#> {xml_nodeset (1)}
#> [1] <h1 id="first">A heading</h1>
```

]

---

## **Strategy**: Use tags and classes to parse html

`source_code`

```html
<html>
<head>
  <title>Page title</title>
</head>
<body>
  <h1 id='first'>A heading</h1>
* <p>Some text &amp; <b>some bold text.</b></p>
  <img src='myimg.png' width='100' height='100'>
</body>
```

]

Use `{rvest}` package to parse html

```r
library(rvest)

html <- read_html(source_code)

html %>% 
* html_elements("p")
```

```
#> {xml_nodeset (1)}
#> [1] <p>Some text &amp; <b>some bold text.</b></p>
```

]

---

## Dealing with multiple nodes (bullet list example)

`source_code`

```html
<ul>
  <li><b>C-3PO</b> is a <i>droid</i> that weighs <span class='weight'>167 kg</span></li>
  <li><b>R4-P17</b> is a <i>droid</i></li>
  <li><b>R2-D2</b> is a <i>droid</i> that weighs <span class='weight'>96 kg</span></li>
  <li><b>Yoda</b> weighs <span class='weight'>66 kg</span></li>
</ul>
```

]

Rendered source code (in a browser)

<ul>
  <li><b>C-3PO</b> is a <i>droid</i> that weighs <span class='weight'>167 kg</span></li>
  <li><b>R4-P17</b> is a <i>droid</i></li>
  <li><b>R2-D2</b> is a <i>droid</i> that weighs <span class='weight'>96 kg</span></li>
  <li><b>Yoda</b> weighs <span class='weight'>66 kg</span></li>
</ul>

]

---

## Dealing with multiple nodes (bullet list example)

`source_code`

]

Use `{rvest}` package to parse html

```r
library(rvest)

html <- read_html(source_code)

html %>% 
* html_elements("li")
```

```
#> {xml_nodeset (4)}
#> [1] <li>\n<b>C-3PO</b> is a <i>droid</i> that weighs <span class="weight">167 kg</span>\n</li>
#> [2] <li>\n<b>R4-P17</b> is a <i>droid</i>\n</li>
#> [3] <li>\n<b>R2-D2</b> is a <i>droid</i> that weighs <span class="weight">96 kg</span>\n</li>
#> [4] <li>\n<b>Yoda</b> weighs <span class="weight">66 kg</span>\n</li>
```

]

---

## Extract the names with `"b"`

`source_code`

]

Use `{rvest}` package to parse html

```r
library(rvest)

html <- read_html(source_code)

html %>% 
  html_elements("li") %>% 
* html_element("b")
```

```
#> {xml_nodeset (4)}
#> [1] <b>C-3PO</b>
#> [2] <b>R4-P17</b>
#> [3] <b>R2-D2</b>
#> [4] <b>Yoda</b>
```

]

---

## Extract the _text_ with `html_text2()`

`source_code`

]

Use `{rvest}` package to parse html

```r
library(rvest)

html <- read_html(source_code)

html %>% 
  html_elements("li") %>% 
  html_element("b") %>% 
* html_text2()
```

```
#> [1] "C-3PO"  "R4-P17" "R2-D2"  "Yoda"
```

]

---

## Extract the weights using `".weight"` class

`source_code`

]

Use `{rvest}` package to parse html

```r
library(rvest)

html <- read_html(source_code)

html %>% 
  html_elements("li") %>% 
* html_element(".weight") %>%
* html_text2()
```

```
#> [1] "167 kg" NA       "96 kg"  "66 kg"
```

]

---

## Putting it together in a data frame

```r
library(rvest)

items <- read_html(source_code) %>% 
  html_elements("li")
```

]

```r
data <- tibble(
  name = items %>% 
    html_element("b") %>% 
    html_text2(), 
  weight = items %>% 
    html_element(".weight") %>% 
    html_text2() %>% 
    parse_number()
)

data
```

```
#> # A tibble: 4 × 2
#>   name   weight
#>   <chr>   <dbl>
#> 1 C-3PO     167
#> 2 R4-P17     NA
#> 3 R2-D2      96
#> 4 Yoda       66
```

]

---

### `html_table()` is awesome (if the site uses an HTML table)

Some pages have HTML tables in the source code, e.g.

https://www.ssa.gov/international/coc-docs/states.html

]

```r
url <- "https://www.ssa.gov/international/coc-docs/states.html"
df <- read_html(url) %>% 
* html_table()

df
```

```
#> [[1]]
#> # A tibble: 56 × 2
#>    X1                   X2   
#>    <chr>                <chr>
#>  1 ALABAMA              AL   
#>  2 ALASKA               AK   
#>  3 AMERICAN SAMOA       AS   
#>  4 ARIZONA              AZ   
#>  5 ARKANSAS             AR   
#>  6 CALIFORNIA           CA   
#>  7 COLORADO             CO   
#>  8 CONNECTICUT          CT   
#>  9 DELAWARE             DE   
#> 10 DISTRICT OF COLUMBIA DC   
#> # … with 46 more rows
```

]

---

## Find elements with [SelectorGadget](https://chrome.google.com/webstore/detail/selectorgadget/mhjhnkcfbdhnjickkkdbjoemdmbfginb?hl=en)

---

## Find elements with "inspect"

---

## Your turn

Scrape data on famous quotes from
http://quotes.toscrape.com/

Your resulting data frame should have these fields:

- `quote`: The quote
- `author`: The author of the quote
- `about_url`: The url to the "about" page

```
#> Rows: 10
#> Columns: 3
#> $ quote     <chr> "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”", "“It is our choices, Harry, that show what we truly are, far more than our abilities.”", "“There are only two w…
#> $ author    <chr> "Albert Einstein", "J.K. Rowling", "Albert Einstein", "Jane Austen", "Marilyn Monroe", "Albert Einstein", "André Gide", "Thomas A. Edison", "Eleanor Roosevelt", "Steve Martin"
#> $ about_url <chr> "http://quotes.toscrape.com/author/Albert-Einstein", "http://quotes.toscrape.com/author/J-K-Rowling", "http://quotes.toscrape.com/author/Albert-Einstein", "http://quotes.toscrape.com/author/Jane-Austen", "http://quotes.toscrape.co…
```

---

# Week 12: .fancy[Webscraping]

### 1. HTML basics
### 1. Scraping static pages
### 2. Scraping multiple pages

### BREAK

### 3. Using APIs

---

# What if there is more than one page to scrape?

<br>

# .orange[Use a loop!]

---

# Iterative scraping!

<br>

## 1. Find the url pattern
## 2. Scrape one page
## 3. Iteratively scrape each page with `map_df()`

---

## 1. Find the url pattern

Example: http://quotes.toscrape.com/

url to page 2: http://quotes.toscrape.com/page/2

Pattern: `http://quotes.toscrape.com/page/` + `#`

<br>

I can _build_ the url to any page with `paste()`:

```r
root <- "http://quotes.toscrape.com/page/"
page <- 3
url <- paste(root, page, sep = "")
url
```

```
#> [1] "http://quotes.toscrape.com/page/3"
```

---

## 2. Scrape one page

Build the url to a single page:

```r
root <- "http://quotes.toscrape.com/page/"
page <- 3
url <- paste(root, page, sep = "")
*url
```

```
#> [1] "http://quotes.toscrape.com/page/3"
```

]

Scrape the data on that page:

```r
*quote_nodes <- read_html(url) %>%
    html_elements(".quote")
df <- tibble(
    quote = quote_nodes %>%
        html_element(".text") %>%
        html_text(),
    author = quote_nodes %>%
        html_element(".author") %>%
        html_text(), 
    about_url = quote_nodes %>%
        html_element("a") %>% 
        html_attr("href")
) %>% 
    mutate(about_url = paste0(url, about_url))
```

]

---

## 3. Iteratively scrape each page with `map_df()`

Make a function to get data from a page:

```r
get_page_data <- function(page) {
    root <- "http://quotes.toscrape.com/page/"
    url <- paste(root, page, sep = "")
    quote_nodes <- read_html(url) %>% 
        html_elements(".quote")
    df <- tibble(
        quote = quote_nodes %>%
            html_element(".text") %>%
            html_text(),
        author = quote_nodes %>%
            html_element(".author") %>%
            html_text(), 
        about_url = quote_nodes %>%
            html_element("a") %>% 
            html_attr("href")
    ) %>% 
        mutate(about_url = paste0(url, about_url))
    return(df)
}
```

]]

Iterate with `map_df()`:

```r
pages <- 1:10

df <- map_df(pages, \(x) get_page_data(x))
```

]]

---

## Your turn

Template code is provided to scrape data on F1 drivers for the 2022 season from
https://www.formula1.com/en/results.html/2022/drivers.html

Your job is to extend it to scrape the data from seasons 2010 to 2022.

Your final dataset should look like this:

```
#> # A tibble: 6 × 8
#>    year position first   last       abb   nationality team                 points
#>   <dbl>    <int> <chr>   <chr>      <chr> <chr>       <chr>                 <int>
#> 1  2022        1 Max     Verstappen VER   NED         Red Bull Racing RBPT    454
#> 2  2022        2 Charles Leclerc    LEC   MON         Ferrari                 308
#> 3  2022        3 Sergio  Perez      PER   MEX         Red Bull Racing RBPT    305
#> 4  2022        4 George  Russell    RUS   GBR         Mercedes                275
#> 5  2022        5 Carlos  Sainz      SAI   ESP         Ferrari                 246
#> 6  2022        6 Lewis   Hamilton   HAM   GBR         Mercedes                240
```

---

# .fancy[Break]

---

# Week 12: .fancy[Webscraping]

### 1. HTML basics
### 1. Scraping static pages
### 2. Scraping multiple pages

### BREAK

### 3. Using APIs

---

# Hopefully you won't need to scrape

---

# Before you start scraping, ask...

<br>

## 1. Is there a formatted dataset I can download?<br>(e.g. see [this page](https://eda.seas.gwu.edu/2022-Fall/help/finding-data.html))

## 2. Is there an API I can use?

---

# .center[Application Programming Interface (API)]

<br>

> A set of defined rules that enable different applications to communicate (and pass data) with each other

<br>

#### .center[Basically, APIs make it easier to get data from the web]

---

# APIs use the `url` to "ask" a website for data

### **Example**: Stock market prices from https://www.alphavantage.co/

API Request:<br>https://www.alphavantage.co/query?function=TIME_SERIES_DAILY_ADJUSTED&symbol={symbol}&apikey={api_key}&datatype=csv

- `function`: The time series of your choice
- `symbol`: Stock price symbol (e.g. `NFLX` = Netflix)
- `apikey`: Your API key (have to register to get one)
- `datatype`: `csv` or `json`

---

## Getting an API key

### 1. Register for a key here: https://www.alphavantage.co/support/#api-key
### 2. Store your key in your `.Renviron`:

```r
usethis::edit_r_environ()
```

### 3. Store your key:

```r
ALPHAVANTAGE_API_KEY={your_key}
```

### 4. Retrieve your key:

```r
api_key <- Sys.getenv("ALPHAVANTAGE_API_KEY")
```

---

## Using your key to get data

```r
api_key <- Sys.getenv("ALPHAVANTAGE_API_KEY")
symbol <- "NFLX" # Netflix

# Build the url data request

url <- paste0(
  "https://www.alphavantage.co/query", 
  "?function=TIME_SERIES_DAILY_ADJUSTED",
  "&symbol=", symbol, 
  "&apikey=", api_key, 
  "&datatype=csv"
)

# Read in the data

df <- read_csv(url)
```

]

```r
glimpse(df)
```

```
#> Rows: 100
#> Columns: 9
#> $ timestamp         <date> 2023-04-06, 2023-04-05, 2023-04-04, 2023-04-03, 2023-03-31, 2023-03-30, 2023-03-29, 2023-03-28, 2023-03-27, 2023-03-24, 2023-03-23, 2023-03-22, 2023-03-21, 2023-03-20, 2023-03-17, 2023-03-16, 2023-03-15, 2023-03-14, 2023-…
#> $ open              <dbl> 339.3400, 345.3000, 348.4900, 341.8300, 340.0500, 340.2699, 326.2900, 326.0600, 327.5500, 320.6300, 304.6800, 306.3100, 306.3200, 299.7900, 310.0600, 304.7500, 292.5100, 295.9700, 287.3400, 297.9050, 312.0800, 309.2900, 31…
#> $ high              <dbl> 340.4800, 345.4300, 349.8000, 348.5800, 345.8400, 343.2900, 332.8500, 333.3200, 336.4400, 331.8300, 322.7800, 306.4500, 307.9200, 307.5000, 310.7600, 316.6000, 306.3100, 297.4500, 299.2400, 298.7900, 312.5100, 311.8300, 31…
#> $ low               <dbl> 332.6300, 336.2526, 343.9500, 340.4001, 337.2001, 335.3000, 325.7300, 321.2800, 324.4100, 320.6300, 304.1400, 293.5400, 300.4300, 296.0000, 300.0000, 301.7100, 292.2800, 290.3100, 285.3301, 289.0000, 294.8800, 305.7500, 30…
#> $ close             <dbl> 339.330, 342.350, 346.750, 348.280, 345.480, 338.430, 332.030, 323.520, 327.660, 328.390, 320.370, 293.900, 305.790, 305.130, 303.500, 310.060, 303.790, 294.940, 293.510, 292.760, 297.780, 311.790, 308.470, 312.030, 315.18…
#> $ adjusted_close    <dbl> 339.330, 342.350, 346.750, 348.280, 345.480, 338.430, 332.030, 323.520, 327.660, 328.390, 320.370, 293.900, 305.790, 305.130, 303.500, 310.060, 303.790, 294.940, 293.510, 292.760, 297.780, 311.790, 308.470, 312.030, 315.18…
#> $ volume            <dbl> 4660542, 4205545, 3298072, 4413650, 5610161, 7131468, 6287336, 6489388, 8625763, 13004085, 15653347, 5808035, 4886279, 5113430, 6918777, 7912371, 9215340, 5956692, 6292385, 5759315, 7443350, 3479474, 4553082, 5660743, 5953…
#> $ dividend_amount   <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ split_coefficient <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …
```

]

---

## Using your key to get data

```r
df %>% 
    ggplot() + 
    geom_line(
      aes(
        x = timestamp, 
        y = adjusted_close
      )
    ) + 
    theme_bw() +
    labs(
        x = "Date",
        y = "Closing Price ($USD)", 
        title = paste0("Stock Prices: ", symbol)
    )
```

]

]

---

# Want something else?

# Read the docs!

## https://www.alphavantage.co/documentation/#dailyadj

---

### Your turn: COVID case data from https://covidactnow.org/

1. Register for a key here: https://apidocs.covidactnow.org/
2. Edit your .Renviron:<br>`usethis::edit_r_environ()`
3. Store your key as `COVID_ACT_NOW_KEY`
4. Load your API key:<br>`api_key <- Sys.getenv("COVID_ACT_NOW_KEY")`
5. Build the url to request historical state-level data
6. Read in the data, then make this figure of daily COVID19 cases in DC

---

# [HW12](https://p4a.seas.gwu.edu/2023-Spring/hw/12-webscraping.html)