-String

str(), | Sequence of characters

-List

list() ["apple", "banana", "pear"], [1, 3, 5, 7] | Sequence of objects/values

-Dictionary

dict(), {} | {'first_name': 'Knute', 'last_name':'Rockne', 'class':'1918'} | Key-value pairs

-Set

set(), {} | {"apple", "banana", "pear"}, {1, 3, 5, 7} | Unordered group of unique values

-Tuple

tuple(), () | ("apple", "banana", "pear"), (1, 3, 5, 7) | Ordered group of values that can include duplicates

-Arrays

An array is a data structure consisting of a collection of elements (values or variables), each identified by at least one array index or key.

-Dictionaries

A mapping between a set of indices (keys) and a set of values. Each key maps to a value, and the association of a key and a value is called a key-value pair.

-Index

In Python, indexing allows us to refer to individual values in specific data structures using their position.

ex. fruits = ["apple", "banana",]

-Iteration

In iteration control structures, a statement or block is executed until the program reaches a certain state, or operations have been applied to every element of a collection.

ex. # list of numbers

numbers = [1, 3, 5, 7, 9, 11, 13, 15, 17]

# sample for loop that iterates over items in list and outputs each number

for number in numbers:

print(number)

-Lists

ex. # create list

fruits = ["apple", "banana", "peach"]

# test if apple is in list

"apple" in fruits # returns true

# test if blueberry is NOT in list

"blueberry" not in fruits # returns false

-Membership Operators

Membership operators are used to test if a sequence is presented in an object

ex. # create list

fruits = ["apple", "banana", "peach"]

# test if apple is in list

"apple" in fruits # returns true

# test if blueberry is NOT in list

"blueberry" not in fruits # returns false

-Sub-lists

ex. # create list with two sub-lists

points = [[0, 1], [2, 3]]

# show list

points

# access first item on list, which is a sublist

points[0]

# access first item WITHIN second item on list

points[1][0]

-Break & Continue

We can exit a loop immediately by using the break statement. break will stop or exit the while loop even if the condition is true. We can skip the rest of the body of a loop and move on to the next iteration using continue.

-Control Flow/Structures

if-then-else (syntax: if, else, elif)

while statements (syntax: while)

functions (built-in and named)

-Enumerate Function

Python function that generates a list of pairs containing each item in the list and its index

-Built in Functions

Functions that are readily available to use.

-Code Blocks

Or compound statement is a lexical structure of source code which is grouped together.

-Docstrings

# function definition

def printThreeTimes():

 '''Prints an input string three times'''

    count = 0 # assign count

    message = input("Type your message here: ") # get input message

    while count < 3: # while statement

        print(message) # print message

        count += 1 # reassign count

-Parameters, Scoping

# assign x variable

x = 1

# function definition

def print_number(x):

 print(x)

# print x variable

print(x)

# call named function

print_number(2)
Content Reflection Week 8:

I think it has been interesting to see the development of possibilities of the computer and how the idea alone was enough to build it; however, no one could have possibly envisioned it would be so life changing. The computer has the ability to affect so many different aspects of our life under a wide range of conditions. In other words, who knew the invention of the computer would make a worldwide industry of post offices almost obsolete. Or what about how now we publish books digitally! You get the point. Computers have pervaded every little aspect of our life and they have changed the course of humanity. Whether that is a good or a bad thing, no one can say for sure. 

So, when reading this article, it was very cool to see that the development of this species altering tool was headlined by a workforce of women. We often envision computer programmers and mathematicians as men. Which reminds me of the movie _____ about the “invention of the computer” by Alan Turing. In this movie, the development of computers would not have been possible without his partner who was a woman. I think this movie does a decent job of portraying the effect that women had on our technology while at the same time explaining the restrictions that society has always put on women in regard to seeing women as innovators.

As I am starting to gain a greater understanding of coding through this week's collaborative assignments, I am at the same time having trouble connecting the pieces we have learned together. I imagine that as I use the fundamental concepts we learned in the past two weeks more, I will continue to have a greater grasp on this new level of abstraction. I look forward to completing this coming week lab and think that because of the sheer number of questions on the lab that I will have adequate chances to shore my understanding. 

Content Reflection Week 9:

I think the goal of building equitable algorithms is very important as we look to the future and the ever growing role that technology plays in our lives. Almost all the knowledge we consume go through the hands of algorithms and filters that change was we see and perceive. Most importantly it forms our decision making and thought processes, so the need for careful consideration of our algorithms is extremely important. On the same page, AI being an increasing factor. AI will continue to take human hands off the wheel and drive society more and more which is extremely scary thought. I hope and pray that the right people will be facing this challenge and focus in particular on creating unbiased AI's with correct morals so that our society can head in the right direction. 

I enjoyed the continued coding lessons and am getting a greater feeling of accomplishment as the coding tasks get harder and more complex. I feel like I am gaining a good understanding of the material. However, I think there should be a space each week to go over certain questions on the labs that people are having trouble with. Perhaps go over those specific lab questions in class and discuss ways to complete the code. I think this would be more worth while than discussing the readings. 

IDE

-An integrated development environment (IDE) is a software application that provides comprehensive facilities to computer programmers for software development. An IDE normally consists of at least a source code editor, build automation tools and a debugger.

Some common IDEs include Replit, Eclipse, Geany, Brackets, Atom, PyCharm, Spyder, RStudio

Anaconda

-An open-source distribution of the Python and R programming languages for scientific computing (data science, machine learning applications, large-scale data processing, predictive analytics, etc.), that aims to simplify package management and deployment. The distribution includes data-science packages suitable for Windows, Linux, and macOS. It is developed and maintained by Anaconda, Inc., which was founded by Peter Wang and Travis Oliphant in 2012

-The Anaconda environment includes a number of specific tools and programs, including:

• JupyterLab

• Jupyter Notebook

• Qt Console

• Spyder

• Glue

• Orange

• RStudio

• Visual Code Studio

Spyder

-Spyder is a free and open source scientific environment written in Python, for Python, and designed by and for scientists, engineers and data analysts. It features a unique combination of the advanced editing, analysis, debugging, and profiling functionality of a comprehensive development tool with the data exploration, interactive execution, deep inspection, and beautiful visualization capabilities of a scientific package

-Spyder's core components include:

• Editor

• IPython console

• Variable explorer

• Plots

• Debugger

• Help

Other Options

• Variable explorer, which lets you see named variables in your program

• Help, which provides additional documentation, information, or resources

• Plots, which will show visualizations generated by your program

• Files, which will show all files currently open or active in your Spyder workspace

The bottom-right pane is the Console, which lets you execute and test Python commands. You can have multiple consoles open simultaneously.

Debugging (Common Python Errors)

• Syntax refers to the structure of a program and rules about that structure. Syntax errors involve things like indentation, parenthesis, etc. The Python interpreter expects the language to be structured a specific way, and throws a syntax error when it's not.

• Runtime errors, or exceptions: appears after a program has started running. These errors indicate something unexpected has happened that interrupts or stops the program execution.

  • Things that can cause a runtime error include:

    • Misspelled or incorrectly capitalized variables or function names

    • Dividing by zero

    • Mismatched data types (i.e. attempting to perform operations on data of the wrong type)

    • Attempting to use a type conversion function on a value that can't be converted

• Semantic: relates to meaning. Semantic errors have to do with the meaning (or purpose/intent) of your code. Semantic errors don't show up as error messages-but the program will not do what you expect it to do.

Stepwise Debugging in Spyder

This mode of debugging includes the following possible steps or "moves":

• Continue (run the program until the next breakpoint)

• Step in (executes the next line of code; if the next line of code is a function, the debugger will 'step into' the first line of that function)

• Step over (executes the next line of code, but will not go line-by-line through the function; the debugger 'steps over' the function code and waits for the function call to return)

• Step out (lets you step out of a function if you used step in and want to 'step out' of the function)

• Stop (stops debugging and terminates the program)

Jupyter

Project Jupyter is a non-profit, open-source project, born out of the IPython Project in 2014 as it evolved to support interactive data science and scientific computing across all programming languages. Jupyter will always be 100% open-source software, free for all to use and released under the liberal terms of the modified BSD license

There are three types of Jupyter notebook cells.

• Code cells allow you to edit and write code with full syntax highlighting and tab completion.

• Markdown cells allow you to document the computational process using descriptive text formatted using the markdown language.

• Raw cells allow you to write output directly. These cells are not evaluated by the notebook and render unmodified. NOTE- not all notebook authoring environments support raw cells

Content Reflection:

The coding lab this week was interesting for a few reasons: one reason was seeing the different IDE that are out there and the possibilities that can arise from each one. I enjoyed seeing the files and plot window that Spyder had integrated into it and can conceive senarios that those tools could be used. Even though I never fully figured out how to use them because when I saved a .py file into Spyder on my computer, the file would not appear when I was in the Spyder application. I also had trouble installing Python packages into Jupyter. This prevented me from completing the final portions of the lab and seeing the effects of the Python packages. Even so, I am excited to learn more about the IDEs and what they have to offer. It is nice to know that before these open-source IDEs were widespread, computer programmers had to have complete knowledge of computer to program anything, because they had to start from scratch, even coding the coding environment itself. 

The second reason I found this weeks expansive explanation and tutorials on IDEs was seeing the advantages and disadvantages of different coding environments. I very much enjoyed the clear and concise route to debugging that Spyder offers, I think that will be a helpful tool in the future along with the easy implementation of files. The files feature offering direct access to the on-board files of my computer simplifies the process of exchanging code between IDEs. On the other hand, I can see the advantages of adding Python packages to Jupyter and the advantages of having pre-coded function in order to help with creating certain types of code. I am sure that simplifies the process tremendously. Finally, I understand the advantages of Replit, offering multitasking and sharing between peers. For group projects this simplifies the process and streamlines group computering. Very similar to how Google Docs allows simultaneous document editing compared to the supreme customization options of Microsoft Word. 

"r" - Read - Opens a file for reading; also the default value

"a" - Append - Opens the file for appending new or additional information; Creates the file if it does not already exist

"w" - Write - Opens the file for writing new information; Creates the file if it does not already exist

"x" - Create - Creates the file if it does not already exist

Type of Data Contained in a File

"t" - Text - Treats file as text data, also the default value

"b" - Binary - Treats the file as binary data

Linear Arrays

In computer science, an array is a data structure consisting of a collection of elements (values or variables), each identified by at least one array index or key...The simplest type of data structure is a linear array, also called one-dimensional array

Associative Arrays

Associative arrays are "an abstract data type that stores a collection of (key, value) pairs, such that each possible key appears at most once in the collection

Delimited File

A delimited file is a sequential file with column delimiters. Each delimited file is a stream of records, which consists of fields that are ordered by column. Each record contains fields for one row. Within each row, individual fields are separated by column delimiters. All fields must be delimited character strings, non-delimited character strings, or external numeric values.

Delimiters

Tab - Delimiter = "/t"

Space - Delimiter = " "

Pipe - Delimiter = "|"

CSV Files

CSV stands for "comma-separated values." CSV files are tabular data structures (i.e. a spreadsheet), stored in a plain-text format. In .csv files, each line represents a row in the spreadsheet, and commas separate cells in each row (thus the file format name comma-separated values).

A few characteristics that distinguish .csv files (or other plain-text delimited data formats) from proprietary spreadsheet file types:

• Columns in a .csv file don't have a value type. Everything is a string.

• Values in a .csv file don't have font or color formatting

• .csv files only contain single worksheets

• .csv files don't store formatting information like cell width/height

• .csv files don't recognize merged cells or other kinds of special formatting (frozen or hidden rows/columns, embedded images, etc.)

open()

The open() function lets us open an existing file or create a new file in Python. For either version of open() (new file or existing file), we need to specify the file name (with the file type extension) and access mode.

write()

Now that we have a newly-created file in Python, we can use the write() function to write content to that file. Let's say we want to create a .txt (plain text) file and write a string to that file. We can do that using write().

Read

We'll also use the csv module to read or load an existing .csv file into Python. The csv module allows us to create a reader object that iterates over lines in a .csv file.

JavaScript Object Notation

JavaScript Object Notation (JSON) is as popular way to format data as a single (purportedly human-readable) string. JavaScript programs use JSON data structures, but we can frequently encounter JSON data outside of a JavaScript environment.

JSON data has the following attributes:

• uses name/value pairs

• separates data using commas

• holds objects using curly braces {}

• holds arrays using square brackets []

Values stored in JSON format must be one of the following data types:

• string

• number

• object (JSON object)

• array

• boolean

• null

Reading JSON into Python

We can read JSON into Python using the json module, which includes a few key functions for loading JSON data into Python:

• json.loads() takes a single string of JSON and loads it as a Python value

• json.load() takes a JSON file (or file-like object) and loads it as a Python value

• json.dumps() takes a Python value and transforms it to a JSON object.

Pandas

a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language

pandas has two main data structures: Series and DataFrame.

• A Series is a one-dimensional, array-like object containing a sequence of values...and an associated array of data labels, called its index

• A DataFrame includes a tabular data structure and contains an ordered collection of columns, each of which can be a different value type

A Series is a one-dimensional, array-like object containing a sequence of values...and an associated array of data labels, called its index. At first glance, a Series looks a lot like a Python list.

A DataFrame includes a tabular data structure "and contains an ordered collection of columns, each of which can be a different value type

Content Reflection:

This week I really enjoyed learning about the ability to import modules. I thought it was cool how you can get third party, pre-coded functions and use them to simplify the coders job. At the same time, it adds another element to coding that I did not expect because each module has a language of its own being able to add delimiters. This added layer of language on top of the already trouble syntax involved in python is very intimidating to me. I can only imagine what its like knowing multiple coding languages and being able to understand every syntax that corresponds to which language. 

Even so, I found the panda module to be particularly helpful in dealing will sets of data because most of the commands that correspond to panda are easy and intuitive to remember. I also enjoy the easy of changing the index of the columns or rows. I presume that I will run into some difficulties when trying to remember that the columns can be accessed through the .column_name whereas the rows have to access using their index value or name using the .loc command, which is not the most intuitive name. 

This weeks reading informed me on the limits and dangers of AI in the future and further reinforced the notion that we must proceed with caution when releasing and testing AI's. Granted there will be growing pains, but with the power and grasp that AI's have on our lives will continue to increase. We as a society must make sure that these AI's are only benefiting our society and not crushing families like the Stipes. I think the best course of action is to create an FDA like entity that will rigorously test AI's that are being created and have decision's by them whether to release it. 

Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.

Load URL and Create a Beautiful Soup Object

from bs4 import BeautifulSoup

import requests

page = requests.get('URL')

soup = BeautifulSoup(page.text, 'html.parser')

HTML Tables

When using BeautifulSoup, we can isolate information on the webpage through interacting with the HTML syntax. HTML uses a few core tags for web pages that include tables.

• table (marks the start and end of a table

• tbody (marks the start and end of the table body)

• tr (marks the start and end of each table row)

• th (marks the start and end of each column in the first row of the table)

• td (marks the start and end of each column after the first row of the table)

Table Structure

<table>

<tr>

<th>First Column Header</th>

<th>Second Column Header</th>

<th>Third Column Header</th>

</tr>

<tr>

<td>Data in first column/first row</td>

<td>Data in second column/first row</td>

<td>Data in third column/first row</td>

</tr>

<tr>

<td>Data in first column/second row</td>

<td>Data in second column/second row</td>

<td>Data in third column/second row</td>

</tr>

</table>

Extract Table

# basic syntax for find

soup.find("HTML TAG GOES HERE")

# a specific example with the p tag

soup.find("p")

or

# basic syntax for find_all, this time using a class instead of a tag

soup.find_all(class_ = "CLASS NAME GOES HERE")

# a specific example using the wikitable sortable class

tables = soup.find_all(class_ = 'wikitable sortable')

#Isolate specific table

table = tables[0]

# get all table rows

rows = table.find_all("tr")

#Iterate over rows in table

table_rows = [] for row in table.find_all('tr'): tags = [] cells = row.find_all(['td', 'th']) tags.append(cells) for cells in tags: try: season = cells[0].contents[0].find('a').contents[0] coach = cells[1].contents[0].find('a').contents[0] location = cells[2].contents[0].find('a').contents[0] opponent = cells[3].contents[0].find('a').contents[0] score = cells[4].contents[0].find('a').contents[0] row_data = [season, coach, location, opponent, score] table_rows.append(row_data) except: continue table_rows # create pandas dataframe df = pd.DataFrame(table_rows, columns = ["Season", "Coach", "Location", "Opponent", "Score"]) # write to csv file df.to_csv('eagles_data.csv', index = False)

Content Reflection:

The reading and topics for this week talk about equitable and fair approach to obtaining, accessing, distributing and interpreting data. The reading presents the ideological solution to reach this end where we should use data as a tool to improve human well-being and not to undermine human dignity. In the reading we learn about a study that was done at Hunter College High School where it was labeled the "saddest place in Manhattan" based on the analysis of tweets surrounding the school. The analysis used key words like sadness to create a frequency map where the school fell. In hindsight, there is obvious reasons why this analysis is fundamentally flawed. Just because people happened to use the word sadness more frequently, does not mean that a specific location in that area is the saddest place around. This study wrongly stigmatized the school and had a negative effect on public opinions of the school and its employees who feel the brunt of this negative publicity. Which was apparent when they interviewed the assistant principal who was publicly displayed and almost a figurehead of the school while having his name besmirched. 

The collaborative problem this week was very interesting because it showed a wide range of possibilities to contort certain lists and raw data to see the data in a different way. Many of these functions like frequency and ngrams allow the user to interpret the data with context surrounding key words that are used in the data. I can see knowledge of these functions as an efficient way to decipher data and prevent misinterpretations that can cause human harm. I think it is important and almost obligatory that as the data scientists of the future we must know these functions and call on them accurately to correctly interpret data so that no unintended consequences and human harm comes from inaccurate data interpretation.