The goal of this assignment is to work with files, iteration, encodings, strings, and string formatting in Python.
You will be doing your work in a Jupyter notebook for this
assignment. You may choose to work on this assignment on a hosted
environment (e.g. tiger)
or on your own local installation of Jupyter and Python. You should use
Python 3.13 for your work. (Older versions may work, but your code will
be checked with Python 3.13.) To use tiger, use the credentials you
received. If you work remotely, make sure to download the .ipynb file to
turn in. If you choose to work locally, Anaconda, miniforge, or uv are probably the easiest ways
to install and manage Python. You can start JupyterLab either from the
Navigator application (anaconda) or via the command-line as
jupyter-lab or jupyter lab.
In this assignment, we will be working with texts from Project Gutenberg which is a library of eBooks which includes a lot of literature for which U.S. copyright has expired. In this assignment, you will process these files, count words, and convert some of the text. We will use Portuguese texts to example the effects of different encodings.
A template notebook is provided with a cell to help download the files given a URL. You will read this data, do some processing, and write a new output file in a similar format. You should write your code so that it can be adapted to work with other texts, but make sure that it runs for the following three texts. Note that the encoding is specified for each.
The assignment is due at 11:59pm on Wednesday, October 15.
You should submit the completed notebook file required for this
assignment on Blackboard. The
filename of the notebook should be a4.ipynb. You
should not turn in the output files as your notebook
should contain the code to create it.
Please make sure to follow instructions to receive full credit. Use a markdown cell to Label each part of the assignment with the number of the section you are completing. You may put the code for each part into one or more cells. Do not use external libraries for reading, parsing, or writing the data files, but you may use the regular expressions module and string methods as necessary. CSCI 490 students do not need to complete Part 4; it is extra credit for them.
The first cell of your notebook should be a markdown cell with a line for your name and a line for your Z-ID. If you wish to add other information (the assignment name, a description of the assignment), you may do so after these two lines.
Use the download_text function in the provided template notebook to download files. Then, write
a function that given the filename and encoding, opens the local file
and reads the downloaded file, returning the processed lines as a list.
Project Gutenberg includes metadata at the beginning of the text and
license information at the end of the text. The actual text is contained
between the lines
*** START OF <ARTICLE> PROJECT GUTENBERG EBOOK <TITLE> ***
and
*** END OF <ARTICLE> PROJECT GUTENBERG EBOOK <TITLE> ***
where <ARTICLE> is "THE" or
"THIS", and <TITLE> is the name or
number of the book. Read (and discard) lines until you hit the
START tag. Then, read in all the lines up to but not including
the END tag into a list of strings. Remove any leading and
trailing whitespace from the lines. You will need to figure out how to
find the START and END tags and which lines should be
included.
strip will be useful to remove whitespace.Next, write a function to loop through the list of lines to find the
set of all unique non-ASCII characters in the text and the subset of
those that are non-alphabetic. Note that you can check whether a
character is in the ASCII subset by checking whether its integer value
is less than 128 (recall ASCII mapped the first 128 characters to the
values 0 through 127). Then, compute the subset of those that are
non-alphabetic characters. Return both the set of
non-ASCII characters and the subset of non-alphabetic, non-ASCII
characters. Test your function with the data from one of the books. For
the first book (17610-8.txt), you should find 22 unique
non-ASCII characters, only two of which are non-alphabetic.
ord function will be useful to obtain the integer
value of a character.string.is... methods may be useful to find
non-alphabetic characters.In the works by Garrett, quotation marks that indicate the start and
end of the quote are used (‘,’,“,”). In addition, two of
the books use the « and » symbols. We wish to
change all of these quotation marks to corresponding non-stylized
versions ' and ". Convert the «
and » symbols to double quotes. Write a method to convert
these quotation marks for the book contents read in in Part 1.
From the output from Part 3, we now wish to change all the
_italicized phrases_ to use the HTML tags <i> and
</i>. Here, any time we see an underscore preceded by
a non-whitespace character and followed by a word boundary, we
want to change it to </i>, and if that is not true
but it is preceded by a word boundary and succeeded by a non-whitespace
character, we want to change it to <i>. Write a
method to convert these character sequences for the book contents read
in in Part 1. If you count the number of <i> and
</i> substrings after conversion, they should be the
same.
\b is a zero-width character class that indicates a
word boundary. It can be useful to test if you have an underscore before
or after a word.As books of poetry, each poem is often identified with a title. In the books we have been looking at, these are indicated with lines that only have capital letters. Note that these lines could also indicate chapters or other headers. However, some roman numerals are also used to separate sections. Write a function to count the number of lines that are headers and the number of lines that are roman numerals in the text. A header is a line written in all capital letters that is not just a roman numeral. You can check whether a line is a roman numeral using the following regular expression from the Regular Expressions Cookbook that assumes you have removed all leading and trailing whitespace first:
r'^(?=[MDCLXVI])M*(C[MD]|D?C{0,3})(X[CL]|L?X{0,3})(I[XV]|V?I{0,3})$'For the file 63438-0.txt, you should find 110 headers
and 105 roman numerals.
Write the converted text (from Part 3 or 4) to a file using the utf-8
encoding. (If you are in CSCI 503, this means applying the functions
from both Parts 3 and 4.) In addition, put your name at the beginning of
the file as the converter, then a line marking the start of the text
(*** START OF THE EBOOK ***), the converted text, and then
a line marking the end of the text
(*** END OF THE EBOOK ***). Finally, use your function from
Part 5 to write the number of headers and roman numerals. The statistics
need to be right-aligned so your output should look like:
From Project Gutenberg (www.gutenberg.org)
Converted by: Iamma Huskie
*** START OF THE EBOOK ***
...
*** END OF THE EBOOK ***
Number of Headers: 110
Number of Roman Numerals: 105The name of the file should be the original filename with
-converted added. For example,
63438-0-converted.txt.
print function calls to write a few data items to
stdout first, then when you are satisfied the format is
correct, write all of the data items to a file.print
function.with statement to make sure all data is written
to the file (or make sure to call close).w flag to open to be
able to write to a file.!cat command:
!cat 63438-0-converted.txt***
lines in Part 6 (5 pts)