Text as data

Where to get text data?

Text is everywhere! And can be a really cool core or supplemental data source. But how do we get text data? One way is through web scraping, through which can collect all sorts of structured and unstructured data for us. Check out this other D-RUG tutorial on web scraping.

Another common way to get text data is through electronic reading of texts, such as reading from PDFs or using OCR on images files. PDFs are usually easier to read and work will with packages like pdftools::, while images can be a bit trickier, even with good OCR functionality like in tesseract::.

Data: Seed Savers Exchange catalogs

The data we’re going to be working with are data from the Seed Savers Exchange catalogs, primarily from 2004. We have these catalogs across several years, including 2015 and 2020, which we have in PDF format. PDFs are easier to work with, and so we can use pdftools for these files.

For example, we can read in a page from the 2015 catalog and see what it looks like

We can read in that page and use the cat function to print it out accounting for page formatting and everything:

text15 <- pdftools::pdf_text("~/Box/quantifying_seed_innovation/sse/sse_census15.pdf")[12]
cat(text15)

                                                                   SOURCES
Bse 2BSeeds, 280 E 1st Ave #1055, Broomfield Co 80038-1055. Email:                  Oregon. We steward rare, diverse and resilient seed varieties for
custserv@2bseeds.com. Phone: (303)465-3395. Offered 183 varieties in                ecologically-minded farmers, gardeners and seed savers. Our seed is
2015, including: 2 Artichoke, 1 Asparagus, 12 Bean, 3 Beet, 1 Bok Choy,             adapted to the Pacific Northwest and other short season northern climates.
1 Broccoli, 1 Brussels Sprouts, 3 Cabbage, 1 Carrot, 1 Celery, 1 Chinese            We sell only public domain, open pollinated seed, as well as many diverse
Cabbage, 2 CHIVES & GARLIC CHIVES, 1 Collard, 1 Cress,                              genepool mixes. All our seed is grown by either us or a few regional friends
10 Cucumber, 1 Eggplant, 1 Endive, 2 Gourd, 2 Kale, 2 Leek, 17 Lettuce,             who help with isolation needs. NONE of our seed are proprietary hybrids
1 Lima Bean, 1 Luffa, 3 Melon, 2 Misc Vegetables, 1 Mizuna, 2 Mustard               (F1), patented, PVP, or genetically modified (GMO). All of our seed is grown
Greens, 3 Okra, 7 ONION COMMON, 8 Pea, 29 Pepper, 8 Radish,                         without chemical fertilizers, herbicides or pesticides, and none of it is
1 Soybean, 1 Spinach, 18 Squash, 2 Swiss Chard, 2 Tomatillo,                        treated. Our farm has been Certified Organic by Oregon Tilth since 2013.
25 Tomato, 3 Watermelon/Citron                                                      Offered 340 varieties in 2015, including: 5 Amaranth, 1 Arugula, 37 Bean,
                                                                                    2 Beet, 1 Bok Choy, 2 Broccoli, 1 Brussels Sprouts, 1 Burdock,
Be1 A Bean Collector's Window Email: upadam@comcast.net. Russ Crow                  2 Cabbage, 1 Carrot, 1 Cauliflower, 1 Celeriac, 1 Chickpea/Garbanzo
is a bean collector from Woodstock, Illinois. He got his start as a member of       Bean, 6 Chicory/Radicchio, 1 Chinese Cabbage, 8 Corn, 1 Cowpea,
John Withee's Wanigan Associates network of bean preservationists in the            2 Cress, 8 Cucumber, 5 Eggplant, 7 Endive, 2 Fava/Broad Bean,
1970s. He is also a long-time member of Seed Savers Exchange. Offered               2 Fennel, 8 Garlic, 1 Ground Cherry, 7 Kale, 3 Leek, 29 Lettuce, 5 Melon,
264 varieties in 2015, including: 247 Bean, 17 Lima Bean                            2 Misc Legumes, 2 Misc Vegetables, 2 Mustard Greens, 1 ONION
                                                                                    BUNCHING, 4 ONION COMMON, 1 ONION MULTIPLIER, 2 Parsley,
                                                                                    1 Parsnip, 4 Pea, 17 Pepper, 1 Purslane, 3 Quinoa, 1 Radish, 1 Runner
Agh1 A Goodly Heritage, 38012 W 196th St, Rayville MO 64084. Email:                 Bean, 3 Rutabaga, 1 Salsify, 1 Scorzonera/Black Salsify, 1 Soybean,
aghfservices@att.net. Phone: (256)590-4727. We are a small family                   2 Spinach, 19 Squash, 4 Sunflower, 1 Tepary Bean, 3 Tomatillo,
operated nursery. We sell natural-grown old varieties of fruit trees, berry         106 Tomato, 1 Turnip, 5 Watermelon/Citron
plants, and grapevines. Retail only. Offered 2 varieties in 2015, including:
2 Garlic
                                                                                    AL24 Albert Lea Seed House , PO Box 127, Albert Lea MN 56007. Phone:
                                                                                    (507)373-3161. Albert Lea Seed began in 1923 as a family-owned seed
Wh5 A.P. Whaley LLC, 1928 Syvrud Rd, Mt Horeb WI 53572. Email:                      store offering field seed and garden seed. Today we are three generations
awhaley@mhtc.net. Phone: (608)437-9081. Offered 337 varieties in 2015,              strong offering knowledge and experience in seed and crop production,
including: 1 Arugula, 31 Bean, 6 Beet, 5 Carrot, 2 Corn, 9 Cucumber,                processing, and cultural practices that benefits any sized farm operation.
7 Eggplant, 1 Ground Cherry, 2 Kale, 15 Lettuce, 3 Lima Bean, 11 Melon,             Offered 163 varieties in 2015, including: 1 Arugula, 1 Asparagus, 14 Bean,
1 Misc Cucurbits, 1 Misc Vegetables, 5 ONION COMMON, 1 ONION                        2 Beet, 1 Bok Choy, 2 Broccoli, 1 Brussels Sprouts, 4 Cabbage, 5 Carrot,
MULTIPLIER, 15 Pea, 26 Pepper, 6 Radish, 4 Runner Bean, 1 Spinach,                  1 Cauliflower, 1 Celery, 8 Corn, 1 Cowpea, 9 Cucumber, 2 Eggplant,
32 Squash, 6 Sunflower, 128 Tomato, 18 Watermelon/Citron                            2 Gourd, 1 Ground Cherry, 1 Kale, 2 Kohlrabi, 4 Lettuce, 1 Lima Bean,
                                                                                    1 Luffa, 4 Melon, 1 Okra, 1 ONION BUNCHING, 3 ONION COMMON,
Ab2 Abbe Hills Seed, L. J. Krouse , 825 Abbe Hills Road, Mt Vernon IA               1 Parsley, 1 Parsnip, 10 Pea, 7 Pepper, 14 Potato, 9 Radish,
52314. Email: lkrouse@abbehills.com. Phone: (319)895-6924. Sells one                1 Rutabaga, 1 Spinach, 24 Squash, 2 Sunflower, 2 Swiss Chard,
variety of Reid's Yellow Dent-type open-pollinated untreated corn for               1 Tomatillo, 7 Tomato, 1 Turnip, 8 Watermelon/Citron
livestock feed or silage. Retail and wholesale. Check www.abbehills.com for
current information. Offered 1 varieties in 2015, including: 1 Corn                 All Allen, Sterling and Lothrop, 191 U.S. Rt. 1, Falmouth ME 04105-1385.
                                                                                    Email: shirley@allensterlinglothrop.com. Phone: (207)781-4142. Fax:
ABB Abbott & Cobb, Inc. , 4151 East Street Rd, Feasterville PA                      (207)781-4143. Specializes in vegetable seeds adapted to northern New
19053-0307. Email: acseed@abbottcobb.com. Phone: (215)245-6666.                     England. Company founded in 1911. Retail only. Offered 122 varieties in
Strictly wholesale! Specializes in sweet corn, watermelon, peppers and              2015, including: 2 Arugula, 12 Bean, 4 Beet, 1 Broccoli, 1 Brussels
squash. Offered 1 varieties in 2015, including: 1 Bean                              Sprouts, 4 Cabbage, 6 Carrot, 1 Cauliflower, 1 Collard, 1 Corn, 1 Cress,
                                                                                    7 Cucumber, 1 Eggplant, 1 Endive, 1 Fava/Broad Bean, 1 Fennel, 3 Kale,
Ada Adams-Briscoe Seed Co., 325 E. Second St. , P.O. Box 19, Jackson                2 Kohlrabi, 1 Leek, 8 Lettuce, 2 Lima Bean, 1 Melon, 2 Misc Vegetables,
GA 30233-0019. Email: abseed@juno.com. Phone: (877)775-7826.                        1 Okra, 1 ONION BUNCHING, 1 ONION COMMON, 2 Parsley, 1 Parsnip,
Specializes in farm, lawn and wildlife seed that is grown in the southeastern       13 Pea, 3 Pepper, 7 Radish, 1 Runner Bean, 1 Rutabaga, 1 Spinach,
U.S. Retail and wholesale. Offered 172 varieties in 2015, including:                16 Squash, 1 Sunflower, 3 Swiss Chard, 3 Tomato, 1 Turnip,
18 Bean, 2 Beet, 1 Broccoli, 5 Cabbage, 3 Carrot, 1 Cauliflower,                    1 Watermelon/Citron, 1 Yard Long Bean
1 Chicory/Radicchio, 4 Collard, 9 Corn, 20 Cowpea, 8 Cucumber,
1 Eggplant, 2 Gourd, 3 Kale, 7 Lettuce, 10 Lima Bean, 4 Melon, 2 Misc               Am1                American                Meadows                   Email:
Legumes, 5 Misc Vegetables, 2 Mustard Greens, 3 Okra, 7 Pea,                        customerservice@americanmeadows.com. Phone: (877)309-7333. Our
7 Pepper, 5 Radish, 1 Rutabaga, 4 Sorghum, 5 Soybean, 1 Spinach,                    approach is simple; for years we have offered gardeners only the best
10 Squash, 3 Sunflower, 7 Tomato, 4 Turnip, 7 Watermelon/Citron                     wildflower seeds, flower bulbs, perennial plants and flowers, vegetable
                                                                                    seeds and flower seed packets; the ones we grow in our own gardens. We
Ad5 Adaptive Seeds, Seed Ambassadors Project , 25079 Brush Creek                    stand ready to help you grow with confidence! Offered 58 varieties in 2015,
Road, Sweet Home OR 97386. Email: seed@adaptiveseeds.com. Phone:                    including: 1 Arugula, 5 Bean, 1 Beet, 1 Broccoli, 2 Cabbage, 1 Carrot,
((541) 367)1105. Adaptive Seeds was established in 2009 by Andrew Still             1 Cauliflower, 2 Corn, 3 Cucumber, 1 Eggplant, 6 Lettuce, 2 Melon,
and Sarah Kleeger, and is based at Open Oak Farm in Sweet Home,                     1 Misc Vegetables, 1 Okra, 1 ONION BUNCHING, 2 Parsley, 6 Pea,

                                                                                1

These data look good. But what about the OCR option? Right outside the DataLab is the 2004 Garden Seed Inventory, but having this book in paper form makes our job more challenging.

Because we don’t have PDFs for these catalogs, the next best option is to treat them as images and OCR them in. For example, I can take an image of each page, then read them in with the tesseract::ocr function:

p25 <- tesseract::ocr('img/p25_plain.png')
cat(p25)

ee
_gisher's Garden Store, PO Box 236, 20750 E, Frontage Road, B,
Fis Titje, MT 59714. Phone: 406-388-6052. Free catalog. Breeders and
era vers of vegetable and flower seeds for high altitudes and short
erswing seasons, including many’ varieties originated by them Een
eyed in Montana in 1923 ae
Florida Mycology Research Center, PO Box 8104 Pen acola, FL
GE 32505. Phone: 850-327-4378. $10 for their main catalog (free if you
mention reading about them in this Garden Seed Inventors), eonigioe
the world’s largest selection of mushroom spores and live cultures
growing supplies and publications. Annual subscription to their
quarterly color mushroom journal, which contains actual [i e mushroom
spore print samples, is $20. Retail and whol esale prices are listed in the
same catalog. . +t
Fo4 Four Seasons Nursery, Division of Plantron, Inc., 17 on
Bloomington, IL 61704. Free catalog. a a
Fo7 Fox Hollow Seed Company, PO Box 148, Me p
Phone: 888-548-7333. Free catalog. Spec
heirloom, untreated varieties, including man e
certified organic. Retail only. €
Fre Fred's Plant Farm, 4589 Ralston Road
800-550-2575. Free catalog. Extensive col
yams. Currently offering nine types, inclu
and wholesale prices are on the same pric
Fun Fungi Perfecti, PO Box 7634, Olympi
9292. Fax: 360-426-9377. Family-own
f _ quality gourmet and medicinal mu
cd chutes for their cultivation. R

Unfortunately, OCRing from an image can be tough. Even with typed font, we’re really not picking up some important pieces of information.

But, Apple / privately developed technology to the rescue. I recently learned that iPhones have some pretty advanced technology for reading text. As I hover over text with an iPhone, I am prompted with some yellow brackets and a text document symbol in the bottom right. I can capture this this by touching that text document symbol and the iPhone will pull out a frame. Then I can use my finger to highlight the text and make sure that iPhone is reading the lines correctly. Last, I can copy that text, paste it in a note, and mail myself a plain text file. So cool!

And after some patience, we can have a whole plain text file. We’ll be using this together:

text <- readtext::readtext("https://raw.githubusercontent.com/d-rug/text-as-data/main/data/sse_seed_inventory_04.txt")

Note what readtext does – it puts all the text in one column and a document id in another.

head(text)

readtext object consisting of 1 document and 0 docvars.
$text
[1] "# A data frame: 1 × 2"                                
[2] "  doc_id                    text               "      
[3] "* <chr>                     <chr>              "      
[4] "1 sse_seed_inventory_04.txt \"\\\"Ab2 Abbe H\\\"...\""

$summary
$summary[[1]]
NULL


attr(,"class")
[1] "trunc_mat"

We’ll just want to worry about the text column, so we can go ahead and subset that.

text <- text[,2]

And now we can glimpse the beginning of the text using the substr function in base R. Looking good.

cat(substr(text, 47548, 51310))

FIS Fisher's Garden Store, 20750 E. Frontage Road, PO
Box 236, Belgrade, MT 59714. Phone: 406-388-6052. Free catalog. Breeders and growers of vegetable and flower seeds for high altitudes and short growing seasons including many
varieties originated by them. Established in Montana in 1923.
Offered 124 non-hybrid vegetable varieties in 2004 including: 1 asparagus, 11 bean, 4 beet, 1 broccoli, 1 brussels sprout, 6 cabbage, 4 carrot, 2 cauliflower, I celery, I Chinese cabbage, 1 collard, 4 com, 1 cress, 5 cucumber, 1 kale, I kohlrabi, 9 lettuce, 1 melon, 1 onion bunching, 5 onion common, 1 onion other, 2 parsley, I parsnip, 8 pea, 3 pepper, 3 potato, 7 radish, 1 runner bean, 1 rutabaga, I salsify, 1 spinach, 16 squash, 1 sunflower ornamental, 1 sunflower edi-ble. 3 Swiss chard, 9 tomato, 2 turnip and 2 miscellaneous.

FI3 Florida Mycology Research Center, P.O. Box 18105,
Pensacola, FL 32523-8105. Phone: 850-327-4378. E-mail:
FloridaMycology@cs.com Web Site: www.mushroomsfmrc com. $10 for their main catalog (free if you mention reading about them in this Garden Seed Inventory), containing the world's largest selection of mushroom spores and live cultures, growing supplies and publications since 1972.
Annual subscription to their quarterly color mushroom joural is $20. Retail and wholesale. Offered 74 mushroom varieties in 2004.

FO4 Four Seasons Nursery, Division of Plantron, Inc.,
1706 Morrissey Drive, Bloomington, IL 61704. Phone: 309-
663-9551. E-mail: customerservice@seasonsnurseries.com.
Web Site: www.4seasonsnurseries.com. Free catalog. Retail only. Offered 1 leek in 2004.

FO7 Fox Hollow Seed Company, 204 Arch Street, Kittan-~~~ OUT OF BUSINESS ning, PA 16201. E-mail: seeds@alltel.net. Web Site: www.foxhollowseed.com. Free catalog. Specializes in open-pollinated, heirloom, untreated varieties including many certified organic vegetable seeds. Retail only. Offered 327
non-hybrid vegetable varieties in 2004 including: 42 bean, 3
beet, 2 broccoli, 1 brussels sprout, 8 cabbage, 1 cardoon, 8 carrot, 2 cauliflower, 1 celeriac, I celery, 3 Chinese cabbage, 4 chicory, 2 collard, 10 com, I comsalad, 3 cress, 7 cucum-ber, 2 eggplant, 2 endive, I fava bean, 1 fennel, 2 kale, 1 kohlrabi, I leek, 23 lettuce, 7 lima, 7 melon, 1 mustard greens, 1 okra, 9 onion common, 2 onion other, 4 parsley, 1 parsnip, 12 pea, 37 pepper, 9 radish, 2 runner bean, rutabaga, 1 salsify, 2 sorghum, 2 spinach, 13 squash, sunflower edible, 2 Swiss chard, 1 tomatillo, 60 tomato, 4 tumip, 7 watermelon, I miscellaneous cucurbitaceae and 8 miscellaneous.

FO13 Forget Me Not Heritage Seed, 729 Erbsville Road,
Waterloo, ON N2J 3Z4, Canada. Web Site: www. forgetmenotseeds.com. On-line catalog. Will ship small quantities of seed to the U.S. for home garden use. Offers untreated, open-pollinated seed grown with respect for the earth. Offered 58 non-hybrid vegetable varieties in 2004
including: 7 bean, 3 beet, I broccoli, 2 cabbage, 2 carrot, 3 cucumber, 4 lettuce. 1 mustard greens, 3 onion common, onion other, 3 pea, 5 pepper, 1 radish, 2 runner bean, spinach, 4 squash, I Swiss chard and 14 tomato.

FRE Fred's Plant Farm, 4589 Ralston Road, Martin, IN
38237. Phone: 800-550-2575. Free catalog. Extensive collection of sweet potatoes and yams, including some old-timers. Retail and wholesale. Offered 9 sweet potato varieties in 2004.

Fun Fungi Perfecti, PO Box 7634, Olympia, WA 98507.
Phone:
360-426-9292. Fax: 360-426-9377. E-mail:
my
comedia@aol.com. Web Site: www.fungi.com. Free bro-chure. $4.95 for 80-page catalog. Family-owned business specializing in high-quality gourmet and medicinal mushroom kits, spawn, plug spawn, along with equipment and techniques for their cultivation. Retail and wholesale.
Offered 16 mushroom varieties in 2004.

So, now let’s get into actually using these data…