動画数：236件
(I love that "Whoops that went too well"-moment at :D)
Read CSV -
Write CSV -
Write TSV -
Read TSV -
Write Excel -
Read Excel - (Start at 6:15 to see installed packages)
Write JSON -
Read JSON -
Write SQL -
# Just in case:If you are using sqlite3 instead of sqlalchemy, you have to use SQL string not the table name:
Read SQL - (Start at 16:59 to see installed packages)
Corey Ception
Ah, I missed the stream :(.. VIM! haha but seriously, I'm a recent convert to vim. It's powerful and efficient.
Youtube premium is ad-free.
I knew that, but...😆
.000 +0200 and i used format='%Y-%m-%d %H:%M:%S.%f %z' It say format does not match. Where did I go wrong. What will the correct format be? Anyone?
for the resampling of the whole DataFrame, say, if I want to get the open price at for the 'open price' of the day and the close price at 23:00:00 for the 'close price' of the day, what function am I supposed to use here?Thank you !
, can you please tell me how to convert this format to standard date and time format ? I try this df['Datetime'] = pd.to_datetime(df['Datetime'], format='%m %d %Y %H:%M'), but it didn't work.... please help.
convert to datetime using to_datetime
Firstly at , we converted all the values of the Series to datetime type from String type using:df[‘Date’] = pd.to_datetime(df[‘Date’], format = ‘%Y-%m-%d %I-%p’)
parse dates while loading data from csv
too complicated
but when doing the same thing while loading csv ,we used a lengthier approach of using a lambda and assigning it to date_parser argument.So my doubt is whether while following the later approach, do we basically apply a function(lambda in this case) to a column('Date') so that the function is applied to each value in that column and it converts each value(String) by calling strptime() ?If so, can we assume to_datetime() method converts a whole Series to datetime type while the later approach converts each value of a Series to datetime object?Is there any way we can replicate the former way while loading the csv file? That look minimal and easy.Thanks in advence
using day_name on whole series using dt class
create column of dayname
min and max methods on datetime series
filtering by dates as string
filter by to_datetime
At time, where you explain filtering the data on basis of start and end date, I tried doing that without changing the date string to datetime format of pandas and it still worked. What you suggested is this "filt=(df['Date']>=pandas.to_datetime('2019-01-01')) & (df['Date']<pandas.to_datetime('2020-01-01'))df.loc[filt]" what I did is filt=(df['Date']>='2019-01-01') & (df['Date']<'2020-01-01')df.loc[filt]. Could you please suggest if the way I did is wrong or will give me the correct results always or not?
set date as index
filter data by just passing the date in brackets
Your tutorials are the bests on the web. Before I can do the index filter ( in the video), I had to sort the dataframe by index(df = df.sort_index()) I don't know why but it was given me an error (AssertionError: <class 'numpy.ndarray'>).Thanks for sharing.
@ df['2019'] does not work for me. Can anyone help me with this issue? ok nevermind i got it figured out. df.index = pd.to_datetime(df.index) then sort it by index if anyone run into the same issueGreat video btw!
using a slice to get specific dates data
Hi all, In minute when filtering df['2020-01':'2020-02] I get an empty dataframe as result. However, if I change the order of the filter df['2020-02':'2020-01] it works. Is there something wrong with this? I didn't sort in any way different than Corey's video but it seems that I need to filter following the pattern of the index to get the correct result. Any comments? Thanks! (Ricardo from Argentina)
calculating average of a slice(timeframe)
getting max value of a column on a given day
df['only_time'] = ''how can i do it without external loop code, because loop taking long time to apply.Thanks for your help.
resampling(breaking down by days) a whole column into a new variable
plotting with matplotlib in pandas
resampling df with multiple columns
using agg to apply different function on different columns while resampling
You could have also used method 'first' for open price within agg functionThanks a lot for these videos.
I got this error 'DataFrame' object has no attribute 'resamlpe'
did the plotting series of tutorials ever come?
the date is same but the time is changing. when i am plotting the graph with this column on x axis it shows 17.07 00..... 17.07 04 i know that at end 00, 04 is the time but i wnt on x axis only time and the dstype of this column is datetime64ns. Can any one help me
when she talks with me xdd
we can use df.replace(['NA', 'Missing'], np.nan, inplace=True) instead
watch from time stamp, Thank me later XD
Question @.So he's saying you can't convert a Nonetype to an int but you can convert a Nonetype to a float? I'm confused here. Can someone please clear this up for me? I didn't really understand Corey's explanation.
I mean if you're coding for more than 50 years, you probably just started as soon as computers fucking came out lol😂😂😂
In of the video. For a one liner code. df['YearsCode'].replace(['Less than 1 year','More than 50 years'],[0,51]), inplace=True). Correct me if I'm wrong I'm new to Python. But great video again Corey! Hats off!
Intro:
Recap:
Basic Aggregate Functions: (`df['col'].mean()`, `df['col'].median()`, `df['col'].mode()`, `df['col'].describe()`, `df.describe()`)- `describe` gives count (# non-NaN rows), mean, std, min, max, and 25%, 50%, and 75% quantiles
Aggregate Column -
median function
Aggregate DataFrame -
describe function
count()
Value Counts -
value_counts()
if you have to do that to a specific country e.g. UK, how many people hobyist from UK and how many are not, how do you do that?
Counts by Percentage: (`df['col'].value_counts(normalize=True)`)
value count in %
Grouping -
Intro to Groupby:- split object, apply function, combine results
grouping the data
Split:df.groupby(['col'])df.get_group(['col_val'])
groupby() function
get_group(), grabbing a specific group by name
doing same by using the filters
Apply Function & Combine:.agg(['mean', 'median'])
using value_counts on filters
at , using .loc u got 20,00 rows and using group by u got 9,000 rows, how??
Small note: at , the more correct syntax would be to select your column within the .loc method: df.loc[filt, 'SocialMedia'].value_counts().
@ How are you able to put the two brackets side-by-side in the line: df.loc[filt]['socialmedia'].value_counts()I dont get why it works with df.loc, but when I tried to set a variable "group" to the groupby function, it doesnt work?
value_counts() for groups
@ How would return just the top row for each country?I want it to look something like this...Country Social MediaAfghanistan Facebook 15Zimbabwe Facebook 3
using loc to find for one country
Corey, at the groupby object forms a series with multindex. I'll try later to use pivot_table() to make index the countries and the columns representing different social apps. Although it won't be tidy data, I think it will be useful to explore it.
percentage by using normalize
median by country group
Multiple Aggregates on Group -
agg function for multiple functions
Gotchas:.apply() on SeriesGroup
People Who Know Python By Country -
using filtering to get python users by country
At in the video, would the following statement be more accurate as we want to know how many people know Python in Indiafilt = df['Country']=='India'df[filt]['LanguageWorkedWith'].str.contains('Python').count()
error on using same approach for groups
apply method to run that on group
Hi Corey. in the group by method in , in the apply() method, the x in the lambda represent the series? in the apply method in the previous lectures isn't the apply method use in every cell/value in a series? Not in the series itself just like the 32:58? I'm new to python. Thanks!
Very interesting tutorial. Thumb up! Particularly at !!
Real-World Problem (Calculation from Several Groups):- What percentage of people from each country know Python?country_respondents = df['Country'].value_counts()country_uses_python = country_grp['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python').sum())result = pd.concat([country_respondents, country_uses_python], axis='columns')result.rename(columns={'Country': 'NumRespondents', 'LanguageWorkedWith': 'NumKnowsPython'})result['PctKnowsPython'] = country_uses_python / country_uses_python * 100result.sort_values(by='PctKnowsPython', ascending=False)result.loc['Japan']
#Ques at , what percentage of people from each country know Python?
Practice Question -
Possible solutions for the question atcountry_grp['LanguageWorkedWith'].apply(lambda series: series.str.contains('Python').sum()/len(series))
try this as the solution to question asked by respected Corey sir at :l_countries = pd.unique(csv.dropna()['Country'])a = [(i, ((gbo.get_group(i)['LanguageWorkedWith'].str.contains('Python').sum()) / (gbo.get_group(i)['LanguageWorkedWith'].count()) * 100)) for i in l_countries]
I was just wondering if on you could have just modified the lambda expression so it could just do all the calculations in place like so:country_grp["LanguageWorkedWith"].apply(lambda country_ser: (country_ser.str.contains("Python").sum()*/country_ser.value_counts().sum())*100)What I tried to do was to count the sum of total respondents for each country and do all the necessary calculations inside the lambda, and, though the result that I got were similar to yours, they still were slightly different.
At since you want to get the percentage of boolean values that are 1 you can pretty much just replace .sum() with .mean(). It's sum/length.
finding the percentage of people using python in each country(group)
. I don’t have the data or code to run to see if this would work right now, but at first glance I’d run:.apply(lambda x: x.str.contains(‘Python’).mean())
I just wanted to share something regarding LanguagesHaveWorked grouped by Country (on ). I think the below code is rather simpler if we just wanna look at the percentages only, but I don't how reliable this method might be
() I added just a short method to get percentage of python users.country_grp['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python').sum())->country_grp['LanguageWorkedWith'].apply(lambda x: (x.str.contains('Python').sum())/x.count())
Concat Series -
using concat for combining series in a dataframe
- U S A ! U S A ! U S A !
Outro: