How to check where datetime is in between two datetimes in pandas

Asked
Active3 hr before
Viewed126 times

4 Answers

pandasdatetime
90%

assign mask to df to return the rows with birth_date between our specified start/end dates,next, set the desired start date and end date to filter df with -- these can be in datetime (numpy and pandas), timestamp, or string format,Get better at data science interviews by solving a few questions per week,We can perform this using a boolean mask First, lets ensure the 'birth_date' column is in date format

Example_snippet/controller/utility/_pandas.js/ import pandas as pd import num. . .
import pandas as pd
import numpy as np
load more v
88%

Determine range time on index or columns value,,Select values at a particular time of the day

Example_snippet/controller/utility/_pandas.js/ >>> i = pd.date_range('2018-04. . .
>>> i = pd.date_range('2018-04-09', periods = 4, freq = '1D20min') >>>
   ts = pd.DataFrame({
      'A': [1, 2, 3, 4]
   }, index = i) >>>
   ts
A
2018 - 04 - 09 00: 00: 00 1
2018 - 04 - 10 00: 20: 00 2
2018 - 04 - 11 00: 40: 00 3
2018 - 04 - 12 01: 00: 00 4
load more v
72%

This solution normally requires start_date, end_date and date column to be datetime format, And in fact, this solution is slow when you are doing a lot of selections by date in a large dataset

Example_snippet/controller/utility/_pandas.js/ df = pd.DataFrame({'date': ['3. . .
df = pd.DataFrame({
   'date': ['3/10/2000', '3/11/2000', '3/12/2000'],
   'value': [2, 3, 4]
}) df['date'] = pd.to_datetime(df['date']) df
Step 2 continued with df = pd.DataFrame({'date': ['3. . .
df = pd.DataFrame({
   'date': ['3/10/2000', '3/11/2000', '3/12/2000'],
   'value': [2, 3, 4]
}) df['date'] = pd.to_datetime(df['date'], dayfirst = True) df
Step 3 continued with df = pd.DataFrame({'date': ['2. . .
df = pd.DataFrame({
   'date': ['2016-6-10 20:30:0', '2016-7-1 19:45:30', '2013-10-12 4:5:1'],
   'value': [2, 3, 4]
}) df['date'] = pd.to_datetime(df['date'], format = "%Y-%d-%m %H:%M:%S") df
Step 4 continued with # Make up 3000 rowsdf = pd.Dat. . .
# Make up 3000 rowsdf = pd.DataFrame({
   'date': ['3/11/2000', '3/12/2000', '3/13/2000'] * 1000
}) % timeit pd.to_datetime(df['date'], infer_datetime_format = True) 100 loops, best of 3: 10.4 ms per loop % timeit pd.to_datetime(df['date'], infer_datetime_format = False) 1 loop, best of 3: 471 ms per loop
Step 5 continued with df = pd.DataFrame({'date': ['3. . .
df = pd.DataFrame({
   'date': ['3/10/2000', 'a/11/2000', '3/12/2000'],
   'value': [2, 3, 4]
}) df['date'] = pd.to_datetime(df['date'])
Step 6 continued with df['date'] = pd.to_datetime(df. . .
df['date'] = pd.to_datetime(df['date'], errors = 'ignore') df
Step 7 continued with df['date'] = pd.to_datetime(df. . .
df['date'] = pd.to_datetime(df['date'], errors = 'coerce') df
Step 8 continued with df = pd.DataFrame({'year': [20. . .
df = pd.DataFrame({
   'year': [2015, 2016],
   'month': [2, 3],
   'day': [4, 5]
}) df['date'] = pd.to_datetime(df) df
Step 9 continued with df = pd.DataFrame({'name': ['T. . .
df = pd.DataFrame({
   'name': ['Tom', 'Andy', 'Lucas'],
   'DoB': ['08-05-1997', '04-28-1996', '12-16-1995']
}) df['DoB'] = pd.to_datetime(df['DoB'])
Step 10 continued with df['year']= df['DoB'].dt.yeard. . .
df['year'] = df['DoB'].dt.yeardf['month'] = df['DoB'].dt.monthdf['day'] = df['DoB'].dt.daydf
Step 11 continued with df['week_of_year'] = df['DoB']. . .
df['week_of_year'] = df['DoB'].dt.weekdf['day_of_week'] = df['DoB'].dt.dayofweekdf['is_leap_year'] = df['DoB'].dt.is_leap_yeardf
Step 12 continued with dw_mapping={ 0: 'Monday', . . .
dw_mapping = {
   0: 'Monday',
   1: 'Tuesday',
   2: 'Wednesday',
   3: 'Thursday',
   4: 'Friday',
   5: 'Saturday',
   6: 'Sunday'
}
df['day_of_week_name'] = df['DoB'].dt.weekday.map(dw_mapping) df
Step 13 continued with today = pd.to_datetime('today'. . .
today = pd.to_datetime('today') df['age'] = today.year - df['DoB'].dt.yeardf
load more v
65%

I have multiple dates in trans_df and master_price, so at the time of joining these two dataframes we will also have to use date? – Neil Dec 3 '18 at 7:48 ,Now, I have master price dataframe from where I am checking whether the price set in trans_df is correct or not basis the transaction date and time is between effective_date_from and effective_date_to in master_price for that particular product ,Connect and share knowledge within a single location that is structured and easy to search,,Software Recommendations

Example_snippet/controller/utility/_datetime.js/ #convert dates with times to d. . .
#convert dates with times to datetimes
master_price['effective_date_from'] = (pd.to_datetime(master_price['effective_date_from'],
      format = '%d-%m-%Y') +
   pd.to_timedelta(master_price['time_from']))
master_price['effective_date_to'] = (pd.to_datetime(master_price['effective_date_to'],
      format = '%d-%m-%Y') +
   pd.to_timedelta(master_price['time_to']))
trans_df['date'] = (pd.to_datetime(trans_df['date'], format = '%d-%m-%Y') +
   pd.to_timedelta(trans_df['time']))

#join together and filter between
df = trans_df.merge(master_price, on = ['code', 'product'], how = 'left')
df = df[df.date.between(df.effective_date_from, df.effective_date_to)]

#add only filterd rows to original
df = trans_df.merge(df, on = ['code', 'product', 'date', 'time'], how = 'left')
cols = ['effective_date_from', 'effective_date_to', 'time_to', 'time_from', 'price_x']
df = df.drop(cols, axis = 1)
#first test missing values then match.mismatch
df['flag'] = np.select([df['price_y'].isnull(),
      df['price_y'] == df['price']
   ],
   [np.nan, 'match'],
   default = 'mismatch')
df = df.rename(columns = {
   'price_y': 'actual_price'
})
print(df)
code price date time product actual_price flag
0 12023 71.23 2018 - 01 - 01 06: 23: 00 06: 23: 00 MS 71.23 match
1 12023 61.00 2018 - 01 - 01 07: 56: 00 07: 56: 00 HS 61.00 match
2 12023 71.23 2018 - 01 - 01 08: 34: 00 08: 34: 00 MS 71.23 match
3 12023 71.30 2018 - 01 - 01 06: 03: 00 06: 03: 00 MS 71.23 mismatch
4 12023 61.00 2018 - 01 - 01 11: 43: 00 11: 43: 00 HS 61.00 match
5 12023 71.23 2018 - 01 - 01 10: 11: 00 10: 11: 00 MS 71.23 match
6 12023 71.23 2018 - 01 - 01 04: 23: 00 04: 23: 00 MS NaN nan
7 12023 72.23 2018 - 01 - 02 10: 11: 00 10: 11: 00 MS 72.23 match
8 12023 72.23 2018 - 01 - 02 04: 23: 00 04: 23: 00 MS 71.23 mismatch