A library for corpus analysis using the Corpus Workbench (CWB) as an efficient back end for indexing and querying large corpora.

polmineR()

Details

The package offers functionality to flexibly create partitions and to carry out basic statistical operations (count, co-occurrences etc.). The original full text of documents can be reconstructed and inspected at any time. Beyond that, the package is intended to serve as an interface to packages implementing advanced statistical procedures. Respective data structures (document term matrices, term co- occurrence matrices etc.) can be created based on the indexed corpora.

A session registry directory (see registry()) combines the registry files for corpora that may reside in anywhere on the system. Upon loading 'polmineR', the files in the registry directory defined by the environment variable CORPUS_REGISTRY are copied to the session registry directory. To see whether the environment variable CORPUS_REGISTRY is set, use the Sys.getenv()-function. Corpora wrapped in R data packages can be activated using the function use().

The package includes a draft shiny app that can be called using polmineR().

Package options

  • polmineR.p_attribute: The default positional attribute.

  • polmineR.left: Default value for left context.

  • polmineR.lineview: A logical value to activate lineview output of kwic().

  • polmineR.pagelength: Maximum number of lines to show when preparing output using DT::datatable() (defaults to 10L).

  • polmineR.meta: Default metadata (s-attributes) to show.

  • polmineR.mc: Whether to use multiple cores.

  • polmineR.cores: Number of cores to use. Passed as argument cl into mclapply().

  • polmineR.browse: Whether to show output in browser.

  • polmineR.buttons: A logical value, whether to display buttons when preparing htmlwidget using DT::datatable().

  • polmineR.specialChars:

  • polmineR.cutoff: Maximum number of characters to display when preparing html output.

  • polmineR.mdsub: A list of pairs of character vectors defining regular expression substitutions applied as part of preprocessing documents for html display. Intended usage: Remove characters that would be misinterpreted as markdown formatting instructions.

  • polmineR.corpus_registry: The system corpus registry directory defined by the environment variable CORPUS_REGISTRY before the polmineR package has been loaded. The polmineR package uses a temporary registry directory to be able to use corpora stored at multiple locations in one session. The path to the system corpus registry directory captures this setting to keep it available if necessary.

  • polmineR.shiny: A logical value, whether polmineR is used in the context of a shiny app. Used to control the apprearance of progress bars depending on whether shiny app is running, or not.

  • polmineR.warn.size: When generating HTML table widgets (e.g. when preparing kwic output to be displayed in RStudio's Viewe pane), the function DT::datatable() that is used internally will issue a warning by default if the object size of the table is greater than 1500000. The warning adresses a client-server scenario that is not applicable in the context of a local RStudio session, so you may want to turn it of. Internally, the warning can be suppressed by setting the option "DT.warn.size" to FALSE. The polmineR option "polmineR.warn.size" is processed by functions calling DT::datatable() to set and reset the value of "DT.warn.size". Please note: The formulation of the warning does not match the scenario of a local RStudio session, but it may still be useful to get a warning when tables are large and slow to process. Therefore, the default value of the setting is FALSE.

References

Jockers, Matthew L. (2014): Text Analysis with R for Students of Literature. Cham et al: Springer.

Baker, Paul (2006): Using Corpora in Discourse Analysis. London: continuum.

Author

Andreas Blaette (andreas.blaette@uni-due.de)

Examples

# The REUTERS corpus included in the RcppCWB package is used in examples
use(pkg = "RcppCWB", corpus = "REUTERS") # activate REUTERS corpus
#> ✔ corpus loaded: REUTERS
r <- corpus("REUTERS")
if (interactive()) show_info(r)

# The package includes GERMAPARLMINI as sample data
use("polmineR") # activate GERMAPARLMINI
#> ✔ corpus loaded: GERMAPARLMINI (version: 0.1.0 | build date: 2023-04-16)
gparl <- corpus("GERMAPARLMINI")
if (interactive()) show_info(gparl)

# Core methods

count("REUTERS", query = "oil")
#>     query count       freq
#>    <char> <int>      <num>
#> 1:    oil    78 0.01925926
count("REUTERS", query = c("oil", "barrel"))
#>     query count        freq
#>    <char> <int>       <num>
#> 1:    oil    78 0.019259259
#> 2: barrel    15 0.003703704
count("REUTERS", query = '"Saudi" "Arab.*"', breakdown = TRUE, cqp = TRUE)
#>               query          match count share
#>              <char>         <char> <int> <num>
#> 1: "Saudi" "Arab.*"   Saudi Arabia     6    50
#> 2: "Saudi" "Arab.*" Saudi Arabia's     3    25
#> 3: "Saudi" "Arab.*"  Saudi Arabian     3    25
dispersion("REUTERS", query = "oil", s_attribute = "id")
#>      query     id count
#>     <char> <char> <int>
#>  1:    oil    127     5
#>  2:    oil    144    12
#>  3:    oil    191     2
#>  4:    oil    194     1
#>  5:    oil    211     0
#>  6:    oil    236     6
#>  7:    oil    237     4
#>  8:    oil    242     2
#>  9:    oil    246     5
#> 10:    oil    248     6
#> 11:    oil    273     5
#> 12:    oil    349     4
#> 13:    oil    352     4
#> 14:    oil    353     4
#> 15:    oil    368     3
#> 16:    oil    489     4
#> 17:    oil    502     5
#> 18:    oil    543     2
#> 19:    oil    704     3
#> 20:    oil    708     1
#>      query     id count
#>     <char> <char> <int>
k <- kwic("REUTERS", query = "oil")
coocs <- cooccurrences("REUTERS", query = "oil")


# Core methods applied to partition

kuwait <- partition("REUTERS", places = "kuwait", regex = TRUE)
#> ... get cpos and strucs
C <- count(kuwait, query = "oil")
D <- dispersion(kuwait, query = "oil", s_attribute = "id")
K <- kwic(kuwait, query = "oil", meta = "id")
#> ... getting corpus positions
#> ... number of hits: 14
#> ... checking that all p-attributes are available
#> ... already present - skip getting ids for p-attribute: word
#> ... generating contexts
CO <- cooccurrences(kuwait, query = "oil")


# Go back to full text

p <- partition("REUTERS", id = 127)
#> ... get cpos and strucs
if (interactive()) read(p)
h <- html(p) %>%
  highlight(highlight = list(yellow = "oil"))
if (interactive()) h_highlighted


# Generate term document matrix (not run by default to save time)
# \donttest{
pb <- partition_bundle("REUTERS", s_attribute = "id")
#> ℹ s-attribute "id" has values: yes
#> ℹ get regions and get values
#> ✔ get regions and get values [6ms]
#> 
#> ℹ instantiate objects (n = 20)
#> ✔ instantiate objects (n = 20) [10ms]
#> 
#> ℹ assign names
#> ✔ assign names [6ms]
#> 
cnt <- count(pb, p_attribute = "word")
tdm <- as.TermDocumentMatrix(cnt, col = "count")
#> ℹ using p-attribute word
#> ℹ create temporary data.table
#> ✔ create temporary data.table ... done
#> 
#> ℹ assign keys to data.table
#> ✔ assign keys to data.table ... done
#> 
#> ℹ make unique keys
#> ✔ make unique keys ... done
#> 
#> ℹ run garbage collection
#> ✔ run garbage collection ... done
#> 
#> ℹ create TermDocumentMatrix
#> ✔ create TermDocumentMatrix ... done
#> 
# }