source("XX_setup.R")
Datasets
This workflow produces many different datasets, some of which are only intermediate, used in the next of the workflow, and some of which are final, summarized and ready-to-use datasets.
Generally speaking, .rds
files (R data file) are used to pass output from one step to another, while preserving variable types (categorical, etc.) as well as nested data (list-columns).
.csv
files are used for final outputs, or where the data being passed between steps is smaller and simpler.
.feather
files are used for the initial conversion of Motus data stored in SQLite tables to a different format. SQLite data bases are great for quick access, but I found that the more filters and joins and other manipulations I applied, the slower the operations became. I found it easier to convert to feather right at the beginning and then use this format for filtering etc. later on.
Data are stored in the Data
folder created by the workflow. This data is too large to be stored on GitHub, so only the workflow scripts and report outputs are online.
Here, we list all the datasets followed by definitions for each variable included.
Data folders
01_Raw
- Downloaded or acquired data
- Motus SQLite databases (
project-XXX.motus
) TagsSpeciesProject.xlsx
from Catherine
02_Datasets
- Intermediate data produced by the workflows
hits
andruns
folders - filtered hit and run level data in feather format [Not included in below descriptions]noise_hits.feather
&noise_runs.feather
- “noisy” runs or hits for later removal [Not included in below descriptions]receivers.csv
- List of receivers and whether they overlap with a species eBird range.transitions_intitial.rds
&bouts_initial.rds
- First pass created transitions and boutstransitions_cleaned.rds
&bouts_cleaned.rds
- Final pass transitions and bouts (but without complete metadata)stn_dist.rds
- Station (groups) distances and minimum flight timesbad_bouts.rds
- List of bouts identified as being ‘bad’ (false positives) to be filtered [Not included in below descriptions]
03_Final
- Final datasets for analysis
bouts_final.csv
- Bouts of activity detected by a station (group) for each bird over timetrans_final.csv
- Transitions between stations (groups) for each bird over timebouts_final_split.csv
- Same asbouts_final.csv
except that bouts which span more than one local day are split at midnight into separate bouts.summary_birds.csv
- Summaries of movement and timing for each birdsummary_daily_bouts.csv
- Daily bouts of activity by a birds at a particular station (group)summary_circadian_bouts.csv
- Subset of the split bouts data including only birds who spent a significant amount of time at a single station. Includes start/end variables for plotting circadian rhythms.summary_stopover_days.csv
- Subset of the daily bouts data showing birds with possible stopovers, identified by those detected at a station (group) over multiple days during migration.summary_stopover_time.csv
- Subset of the daily bouts data showing birds with possible stopovers, identified by those spending time at at a station during migration.
Data variables
Data/02_Datasets/bouts_cleaned.rds
Data/02_Datasets/bouts_initial.rds
Data/02_Datasets/receivers.csv
Data/02_Datasets/stn_dist.rds
Data/02_Datasets/transitions_cleaned.rds
Data/02_Datasets/transitions_initial.rds
Data/03_Final/bouts_final.csv
Data/03_Final/bouts_final_split.csv
Data/03_Final/summary_birds.csv
Data/03_Final/summary_circadian_bouts.csv
Data/03_Final/summary_daily_bouts.csv
Data/03_Final/summary_stopover_days.csv
Data/03_Final/summary_stopover_time.csv
Data/03_Final/transitions_final.csv
02_Datasets/bouts_cleaned.rds
Data/02_Datasets/bouts_cleaned.rds | ||
Variable | Description | Example Value |
---|---|---|
tagDeployID |
ID of the tag/individual from Motus |
2661 |
stn_group |
Receiver group. Can be one of a pair in transitions ( |
1113-1115 |
bout_id |
Unique identifier of bouts within a tag at a particular station group |
1 |
timeBegin |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-10 12:22:18.9287 |
timeEnd |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-10 12:26:26.2143 |
runID |
A list of runIDs summarized into this bout (nested list column) |
622034690:622034692 |
n_runs |
The number of runs summarized into this bout |
3 |
len |
The |
c(7, 8, 7) |
ant |
The |
c(“2”, “1”, “3”) |
total_time |
The duration in minutes of the bout |
4.12142666578293 |
len_min |
The min/max |
7 |
len_max |
The min/max |
8 |
n_ant |
The number of unique antennas summarized into this bout |
3 |
dateBegin |
The start/end dates of the bout |
2014-10-10 |
recvDeployID |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
1113 |
id |
Unique identifier for a bout ( |
2661_1113-1115_1 |
recvDeployLat |
Reciever station coordinates from Motus |
48.2581 |
recvDeployLon |
Reciever station coordinates from Motus |
-69.5214 |
02_Datasets/bouts_initial.rds
Data/02_Datasets/bouts_initial.rds | ||
Variable | Description | Example Value |
---|---|---|
tagDeployID |
ID of the tag/individual from Motus |
2661 |
stn_group |
Receiver group. Can be one of a pair in transitions ( |
1113-1115 |
bout_id |
Unique identifier of bouts within a tag at a particular station group |
1 |
timeBegin |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-10 12:22:18.9287 |
timeEnd |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-10 12:26:26.2143 |
runID |
A list of runIDs summarized into this bout (nested list column) |
622034690:622034692 |
n_runs |
The number of runs summarized into this bout |
3 |
len |
The |
c(7, 8, 7) |
ant |
The |
c(“2”, “1”, “3”) |
total_time |
The duration in minutes of the bout |
4.12142666578293 |
len_min |
The min/max |
7 |
len_max |
The min/max |
8 |
n_ant |
The number of unique antennas summarized into this bout |
3 |
dateBegin |
The start/end dates of the bout |
2014-10-10 |
recvDeployID |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
1113 |
id |
Unique identifier for a bout ( |
2661_1113-1115_1 |
recvDeployLat |
Reciever station coordinates from Motus |
48.2581 |
recvDeployLon |
Reciever station coordinates from Motus |
-69.5214 |
02_Datasets/receivers.csv
Data/02_Datasets/receivers.csv | ||
Variable | Description | Example Value |
---|---|---|
recvDeployID |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
629 |
recvDeviceID |
Motus receiver device ID (not used) |
337 |
name |
Motus receiver deployment name (not used) |
Werden |
species_code |
eBird species code |
amered |
in_range |
Logical value indicating whether a receiver is within a given species natural range |
TRUE |
id |
Unique identifier for a bout ( |
16890 |
english |
English species name |
American Redstart |
problem |
Whether or not there is an issue with that receiver, such as ‘mobile station’, ‘test station’, ‘out of all species ranges’, ‘missing coords’, or ‘out of americas’ |
none |
02_Datasets/stn_dist.rds
Data/02_Datasets/stn_dist.rds | ||
Variable | Description | Example Value |
---|---|---|
recv1 |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
645 |
recv2 |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
645 |
next_dist |
Distance in Kilometers to the next station in a transition or station pair |
0 |
recv_pair |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
645_645 |
min_time |
The minimum about of time in hours required to travel to the next station to avoid being flagged as too fast |
0s |
02_Datasets/transitions_cleaned.rds
Data/02_Datasets/transitions_cleaned.rds | ||
Variable | Description | Example Value |
---|---|---|
tagDeployID |
ID of the tag/individual from Motus |
4350 |
id1 |
Unique identifier for a bout ( |
4350_1115_1 |
id2 |
Unique identifier for a bout ( |
4350_701_1 |
overlap |
Whether or not a bout overlaps another for that individual |
FALSE |
stn1 |
Receiver group. Can be one of a pair in transitions ( |
1115 |
b1 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-09 15:00:20.7792 |
e1 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-09 15:02:42.1372 |
recv1 |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
1115 |
lat1 |
Reciever station coordinates from Motus |
48.1232 |
lon1 |
Reciever station coordinates from Motus |
-69.7168 |
stn2 |
Receiver group. Can be one of a pair in transitions ( |
701 |
b2 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-31 14:06:40.4045 |
e2 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-31 14:16:05.2456 |
recv2 |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
701 |
lat2 |
Reciever station coordinates from Motus |
47.0794 |
lon2 |
Reciever station coordinates from Motus |
-64.8736 |
recv_pair |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
1115_701 |
next_dist |
Distance in Kilometers to the next station in a transition or station pair |
381.149431206303 |
min_time |
The minimum about of time in hours required to travel to the next station to avoid being flagged as too fast |
5293.74210008754s (~1.47 hours) |
trans_id |
Unique identified of transitions |
1 |
lag1 |
The amount of time in minutes between the end of the first bout and the beginning of the next (lag1) or the end of the next (lag2). Both lags are accounted for to deal with overlapping bouts. |
31623.9711216648 |
lag2 |
The amount of time in minutes between the end of the first bout and the beginning of the next (lag1) or the end of the next (lag2). Both lags are accounted for to deal with overlapping bouts. |
31633.3851399978 |
time_diff |
The minimum amount of time in hours from the end of the first bout to either the start or end of the next (to account for overlapping bouts |
1897438.26729989s (~3.14 weeks) |
speed |
The speed of travel between stations, |
0.200875800691366 |
mid_date |
The date at the midpoint of the transition |
2014-10-20 |
migration |
If both start and end of the transition occur within October to December, we call this a ‘south’ migration. If both the start and end of the transition occur within February to July, we call this a ‘north’ migration. Used for assessing incorrect directions in transitions. |
south |
connected |
From Amie’s scripts, if the transition speed is greater than 5 m/s |
FALSE |
02_Datasets/transitions_initial.rds
Data/02_Datasets/transitions_initial.rds | ||
Variable | Description | Example Value |
---|---|---|
tagDeployID |
ID of the tag/individual from Motus |
4350 |
id1 |
Unique identifier for a bout ( |
4350_1115_1 |
id2 |
Unique identifier for a bout ( |
4350_701_1 |
overlap |
Whether or not a bout overlaps another for that individual |
FALSE |
stn1 |
Receiver group. Can be one of a pair in transitions ( |
1115 |
b1 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-09 15:00:20.7792 |
e1 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-09 15:02:42.1372 |
recv1 |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
1115 |
lat1 |
Reciever station coordinates from Motus |
48.1232 |
lon1 |
Reciever station coordinates from Motus |
-69.7168 |
stn2 |
Receiver group. Can be one of a pair in transitions ( |
701 |
b2 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-31 14:06:40.4045 |
e2 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-31 14:16:05.2456 |
recv2 |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
701 |
lat2 |
Reciever station coordinates from Motus |
47.0794 |
lon2 |
Reciever station coordinates from Motus |
-64.8736 |
recv_pair |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
1115_701 |
next_dist |
Distance in Kilometers to the next station in a transition or station pair |
381.149431206303 |
min_time |
The minimum about of time in hours required to travel to the next station to avoid being flagged as too fast |
5293.74210008754s (~1.47 hours) |
trans_id |
Unique identified of transitions |
1 |
lag1 |
The amount of time in minutes between the end of the first bout and the beginning of the next (lag1) or the end of the next (lag2). Both lags are accounted for to deal with overlapping bouts. |
31623.9711216648 |
lag2 |
The amount of time in minutes between the end of the first bout and the beginning of the next (lag1) or the end of the next (lag2). Both lags are accounted for to deal with overlapping bouts. |
31633.3851399978 |
time_diff |
The minimum amount of time in hours from the end of the first bout to either the start or end of the next (to account for overlapping bouts |
1897438.26729989s (~3.14 weeks) |
speed |
The speed of travel between stations, |
0.200875800691366 |
mid_date |
The date at the midpoint of the transition |
2014-10-20 |
migration |
If both start and end of the transition occur within October to December, we call this a ‘south’ migration. If both the start and end of the transition occur within February to July, we call this a ‘north’ migration. Used for assessing incorrect directions in transitions. |
south |
connected |
From Amie’s scripts, if the transition speed is greater than 5 m/s |
FALSE |
03_Final/bouts_final.csv
Data/03_Final/bouts_final.csv | ||
Variable | Description | Example Value |
---|---|---|
tagDeployID |
ID of the tag/individual from Motus |
2661 |
stn_group |
Receiver group. Can be one of a pair in transitions ( |
1113-1115 |
bout_id |
Unique identifier of bouts within a tag at a particular station group |
1 |
timeBegin |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-10 12:22:18 |
timeEnd |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-10 12:26:26 |
n_runs |
The number of runs summarized into this bout |
3 |
total_time_min |
The duration in minutes of the bout |
4.12142666578293 |
len_min |
The min/max |
7 |
len_max |
The min/max |
8 |
n_ant |
The number of unique antennas summarized into this bout |
3 |
dateBegin |
The start/end dates of the bout |
2014-10-10 |
recvDeployID |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
1113 |
id |
Unique identifier for a bout ( |
2661_1113-1115_1 |
recvDeployLat |
Reciever station coordinates from Motus |
48.2581 |
recvDeployLon |
Reciever station coordinates from Motus |
-69.5214 |
speciesID |
ID of the species associated with that tagDeployID from Motus |
20420 |
english |
English species name |
Pine Siskin |
utc_offset_h |
The number of hours the local time zone of that receiver is offset from UTC |
-4 |
timeBeginLocal |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-10 08:22:18 |
timeEndLocal |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-10 08:26:26 |
03_Final/bouts_final_split.csv
Data/03_Final/bouts_final_split.csv | ||
Variable | Description | Example Value |
---|---|---|
english |
English species name |
Pine Siskin |
tagDeployID |
ID of the tag/individual from Motus |
2661 |
stn_group |
Receiver group. Can be one of a pair in transitions ( |
1113-1115 |
dateBeginLocal |
The date over which the bouts have been summarized (corresponds to the Local timezone for the receiver or receiver group) |
2014-10-10 |
time_hrs |
The number of hours a bird was recorded at that receiver or receiver group on that date |
0.0686904444297155 |
recvDeployLat |
Reciever station coordinates from Motus |
48.2581 |
recvDeployLon |
Reciever station coordinates from Motus |
-69.5214 |
03_Final/summary_birds.csv
Data/03_Final/summary_birds.csv | ||
Variable | Description | Example Value |
---|---|---|
english |
English species name |
Pine Siskin |
tagDeployID |
ID of the tag/individual from Motus |
2661 |
travelled |
Whether or not individuals moved more than 100km |
TRUE |
migrated |
Whether or not individuals moved more than 1 latitude |
TRUE |
migrated_far |
Whether or not individuals moved more than 5 latitude |
FALSE |
n_stn |
Number of station groups visited by an individual |
3 |
min_date |
First date an individual was detected |
2014-10-10 |
mean_date |
Average date of bounts for an individual |
2014-10-22 |
max_date |
Final date an individual was detected |
2014-10-28 |
total_time_hrs |
Total amount of time in bouts an individual was detected |
1.41324997213152 |
mean_time_hrs |
Average amount of time per bout an individual was detected |
0.282649994426303 |
res_stn |
First station group an individual was detected at, this is considered the ‘resident’ station |
1113-1115 |
mean_time_no_resident_hrs |
Average amount of time per bout excluding the resident station an individual was detectect |
0.284353787015986 |
first_time_hrs |
Total amount of time an individual was detected at the resident station (first station) |
0.560188611083561 |
last_time_hrs |
Total amount of time an individual was detected at the last station in its record |
0.0490849722094006 |
03_Final/summary_circadian_bouts.csv
Data/03_Final/summary_circadian_bouts.csv | ||
Variable | Description | Example Value |
---|---|---|
tagDeployID |
ID of the tag/individual from Motus |
48775 |
stn_group |
Receiver group. Can be one of a pair in transitions ( |
8759-8766 |
bout_id |
Unique identifier of bouts within a tag at a particular station group |
1 |
n_runs |
The number of runs summarized into this bout |
1 |
total_time_min |
The duration in minutes of the bout |
36.4333333333333 |
len_min |
The min/max |
23 |
len_max |
The min/max |
23 |
n_ant |
The number of unique antennas summarized into this bout |
1 |
recvDeployID |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
8759 |
id |
Unique identifier for a bout ( |
48775_8759-8766_1 |
recvDeployLat |
Reciever station coordinates from Motus |
49.1765 |
recvDeployLon |
Reciever station coordinates from Motus |
-109.5086 |
speciesID |
ID of the species associated with that tagDeployID from Motus |
14040 |
english |
English species name |
Horned Lark |
utc_offset_h |
The number of hours the local time zone of that receiver is offset from UTC |
-6 |
timeBeginLocal |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2023-07-31 18:25:11 |
timeEndLocal |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2023-07-31 19:01:37 |
dateBeginLocal |
The date over which the bouts have been summarized (corresponds to the Local timezone for the receiver or receiver group) |
2023-07-31 |
year |
The year of the |
2023 |
start |
The daily start/end times of a bout in decimal hours. For example, 2023-10-01 10:45:01 would have a decimal hour time of 10.75. This is good for plotting daily activity rhythms |
18.4166666666667 |
end |
The daily start/end times of a bout in decimal hours. For example, 2023-10-01 10:45:01 would have a decimal hour time of 10.75. This is good for plotting daily activity rhythms |
19.0166666666667 |
03_Final/summary_daily_bouts.csv
Data/03_Final/summary_daily_bouts.csv | ||
Variable | Description | Example Value |
---|---|---|
english |
English species name |
Pine Siskin |
tagDeployID |
ID of the tag/individual from Motus |
2661 |
stn_group |
Receiver group. Can be one of a pair in transitions ( |
1113-1115 |
dateBeginLocal |
The date over which the bouts have been summarized (corresponds to the Local timezone for the receiver or receiver group) |
2014-10-10 |
time_hrs |
The number of hours a bird was recorded at that receiver or receiver group on that date |
0.0686904444297155 |
recvDeployLat |
Reciever station coordinates from Motus |
48.2581 |
recvDeployLon |
Reciever station coordinates from Motus |
-69.5214 |
03_Final/summary_stopover_days.csv
Data/03_Final/summary_stopover_days.csv | ||
Variable | Description | Example Value |
---|---|---|
english |
English species name |
Pine Siskin |
tagDeployID |
ID of the tag/individual from Motus |
2661 |
stn_group |
Receiver group. Can be one of a pair in transitions ( |
1111 |
dateBeginLocal |
The date over which the bouts have been summarized (corresponds to the Local timezone for the receiver or receiver group) |
2014-10-24 |
time_hrs |
The number of hours a bird was recorded at that receiver or receiver group on that date |
0.519623333281941 |
recvDeployLat |
Reciever station coordinates from Motus |
49.0487 |
recvDeployLon |
Reciever station coordinates from Motus |
-68.4659 |
03_Final/summary_stopover_time.csv
Data/03_Final/summary_stopover_time.csv | ||
Variable | Description | Example Value |
---|---|---|
english |
English species name |
Blue-headed Vireo |
tagDeployID |
ID of the tag/individual from Motus |
50721 |
stn_group |
Receiver group. Can be one of a pair in transitions ( |
8934 |
dateBeginLocal |
The date over which the bouts have been summarized (corresponds to the Local timezone for the receiver or receiver group) |
2023-10-18 |
time_hrs |
The number of hours a bird was recorded at that receiver or receiver group on that date |
0.507262722253799 |
recvDeployLat |
Reciever station coordinates from Motus |
40.4107 |
recvDeployLon |
Reciever station coordinates from Motus |
-76.0796 |
03_Final/transitions_final.csv
Data/03_Final/transitions_final.csv | ||
Variable | Description | Example Value |
---|---|---|
tagDeployID |
ID of the tag/individual from Motus |
4350 |
id1 |
Unique identifier for a bout ( |
4350_1115_1 |
id2 |
Unique identifier for a bout ( |
4350_701_1 |
overlap |
Whether or not a bout overlaps another for that individual |
FALSE |
stn1 |
Receiver group. Can be one of a pair in transitions ( |
1115 |
b1 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-09 15:00:20 |
e1 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-09 15:02:42 |
recv1 |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
1115 |
lat1 |
Reciever station coordinates from Motus |
48.1232 |
lon1 |
Reciever station coordinates from Motus |
-69.7168 |
stn2 |
Receiver group. Can be one of a pair in transitions ( |
701 |
b2 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-31 14:06:40 |
e2 |
Begining and ending times for bouts. Can be one of a pair in transitions ( |
2014-10-31 14:16:05 |
recv2 |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
701 |
lat2 |
Reciever station coordinates from Motus |
47.0794 |
lon2 |
Reciever station coordinates from Motus |
-64.8736 |
recv_pair |
Individial receiver stations (recvDeployID from Motus). Can be one of a pair in transitions ( |
1115_701 |
next_dist |
Distance in Kilometers to the next station in a transition or station pair |
381.149431206303 |
trans_id |
Unique identified of transitions |
1 |
mid_date |
The date at the midpoint of the transition |
2014-10-20 |
speciesID |
ID of the species associated with that tagDeployID from Motus |
20420 |
english |
English species name |
Pine Siskin |
min_time_hrs |
The minimum about of time in hours required to travel to the next station to avoid being flagged as too fast |
5293.74210008754 |
time_diff_hrs |
The minimum amount of time in hours from the end of the first bout to either the start or end of the next (to account for overlapping bouts |
1897438.26729989 |
next_dist_km |
Distance in Kilometers to the next station in a transition or station pair |
381.149431206303 |
lag1_min |
The amount of time in minutes between the end of the first bout and the beginning of the next (lag1) or the end of the next (lag2). Both lags are accounted for to deal with overlapping bouts. |
31623.9711216648 |
lag2_min |
The amount of time in minutes between the end of the first bout and the beginning of the next (lag1) or the end of the next (lag2). Both lags are accounted for to deal with overlapping bouts. |
31633.3851399978 |
speed_m_s |
The speed of travel between stations, |
0.200875800691366 |