TDM 10200: Python Project 5 — Spring 2025

Motivation: Real world data has a lot of missing data. It is also helpful to be able to take a subset of data.

Context: It is worthwhile to be prepared to have missing data and to know how to work with it.

Scope: Dealing with missing data, and taking subsets of data.

Learning Objectives:

  • Learning about how to work with missing data and how to take subsets of data.

Make sure to read about, and use the template found here, and the important information about project submissions here.

Dataset(s)

This project will use the following datasets:

  • /anvil/projects/tdm/data/death_records/DeathRecords.csv

  • /anvil/projects/tdm/data/8451/The_Complete_Journey_2_Master/5000_transactions.csv

  • /anvil/projects/tdm/data/beer/reviews_sample.csv

  • /anvil/projects/tdm/data/election/itcont1980.txt

  • /anvil/projects/tdm/data/flights/subset/1990.csv

Questions

Example 1:

Example 2:

Example 3:

Example 4:

Example 5:

Question 1 (2 pts)

In the death records file:

/anvil/projects/tdm/data/death_records/DeathRecords.csv

  1. Build a subset of the data for which Sex=='F' and check the head of the subset to make sure that you only have 'F' values in the Sex column of your subset.

  2. Make a table of the Age values from the subset of female data in question 1a. Put the values in order by Age using sort_index() instead of sort_values(). Then plot the table of these Age values. (Notice that 999 is used when the Age value is missing in part 1b!)

  3. Now revise your subset from question 1a, so that you build a subset of the data for which Sex=='F' & Age!=999 and then make of table of the Age values from this revised subset of female data. Put these values in order by Age using sort_index() instead of sort_values() (in particular, the age values of 999 should be removed now), and plot the table of these Age values.

Deliverables

  • a. The head of the subset of data for which Sex=='F'

  • b. Plot of the table of Age values for the subset in 1a.

  • c. Revise questions 1a and 1b so that Sex=='F' & Age!=999

Question 2 (2 pts)

In the grocery store file:

/anvil/projects/tdm/data/8451/The_Complete_Journey_2_Master/5000_transactions.csv

there are more than 10 million lines of data, as we can see if we check myDF.shape. Each line corresponds to the purchase of an item. The SPEND column is negative when a purchase is refunded, i.e., the item is returned and the money is given back to the customer.

Create a smaller data set called refundsDF that contains only the lines of data for which the SPEND column is negative. Make a table of the STORE_R values in this refundsDF subset, and show the number of times that each STORE_R value appears in the refundsDF subset.

When you read in the grocery data, you need to assign column names, as follows:

myDF = pd.read_csv("/anvil/projects/tdm/data/8451/The_Complete_Journey_2_Master/5000_transactions.csv")

myDF.columns = ["BASKET_NUM", "HSHD_NUM", "PURCHASE", "PRODUCT_NUM", "SPEND", "UNITS", "STORE_R", "WEEK_NUM", "YEAR"]

Deliverables

  • Show the number of refunds for each STORE_R value in the refundsDF subset. (For instance, CENTRAL stores had 2750 refunds.)

Question 3 (2 pts)

In this file of beer reviews /anvil/projects/tdm/data/beer/reviews_sample.csv

Make a subset of the beers that have (score != 5) & (overall == 5) (in other words the score value is not equal to 5 but the overall value is equal to 5). How many lines of data are in this subset?

Deliverables

  • How many lines of data are in the subset that has (score != 5) & (overall == 5) ?

Question 4 (2 pts)

Read in the 1980 election data using:

/anvil/projects/tdm/data/election/itcont1980.txt

Remember that we need to specify the column names for the 1980 election donation data:

myDF = pd.read_csv("/anvil/projects/tdm/data/election/itcont1980.txt", header=None, sep='|')

myDF.columns = ["CMTE_ID", "AMNDT_IND", "RPT_TP", "TRANSACTION_PGI", "IMAGE_NUM", "TRANSACTION_TP", "ENTITY_TP", "NAME", "CITY", "STATE", "ZIP_CODE", "EMPLOYER", "OCCUPATION", "TRANSACTION_DT", "TRANSACTION_AMT", "OTHER_ID", "TRAN_ID", "FILE_NUM", "MEMO_CD", "MEMO_TEXT", "SUB_ID"]

There are only 9 entries in which the TRANSACTION_DT value is missing, namely: one donation from CURCIO, BARBARA G and two donations from WOLFF, GARY W. and six donations from who?? (find their identity)! Find the name of the person who made six donations in 1980 with a missing TRANSACTION_DT.

Deliverables

  • Find the name of the person who made 6 donations in 1980 with a missing TRANSACTION_DT.

Question 5 (2 pts)

Consider the 1990 flight data:

/anvil/projects/tdm/data/flights/subset/1990.csv

This data set has information about 5270893 flights.

  1. For how many flights is the DepDelay missing and also (simultaneously) the ArrDelay is missing too?

  2. For how many flights is the DepDelay given but the ArrDelay is missing?

  3. For how many flights is the ArrDelay given but the DepDelay is missing?

Deliverables

  • a. Find the number of flights for which the DepDelay is missing and also (simultaneously) the ArrDelay is missing too.

  • b. Find the number of flights for which the DepDelay is given but the ArrDelay is missing.

  • c. Find the number of flights for which the ArrDelay is given but the DepDelay is missing.

Submitting your Work

Please make sure that you added comments for each question, which explain your thinking about your method of solving each question. Please also make sure that your work is your own work, and that any outside sources (people, internet pages, generating AI, etc.) are cited properly in the project template.

If you have any questions or issues regarding this project, please feel free to ask in seminar, over Piazza, or during office hours.

Prior to submitting your work, you need to put your work into the project template, and re-run all of the code in your Jupyter notebook and make sure that the results of running that code is visible in your template. Please check the detailed instructions on how to ensure that your submission is formatted correctly. To download your completed project, you can right-click on the file in the file explorer and click 'download'.

Once you upload your submission to Gradescope, make sure that everything appears as you would expect to ensure that you don’t lose any points.

Items to submit

  • firstname_lastname_project5.ipynb

It is necessary to document your work, with comments about each solution. All of your work needs to be your own work, with citations to any source that you used. Please make sure that your work is your own work, and that any outside sources (people, internet pages, generating AI, etc.) are cited properly in the project template.

You must double check your .ipynb after submitting it in gradescope. A very common mistake is to assume that your .ipynb file has been rendered properly and contains your code, markdown, and code output even though it may not.

Please take the time to double check your work. See here for instructions on how to double check this.

You will not receive full credit if your .ipynb file does not contain all of the information you expect it to, or if it does not render properly in Gradescope. Please ask a TA if you need help with this.