Data Retrieval

Here's suggested pseudo-code for downloading Firehose Data

Set current_epoch_timestamp = get_currect_epoch_timestamp()
Read list of zip files from the firehose link
Download zip file and extract
Every 1 minute:
Read Firehose Link
For each zip file name(*):
    If Number(zip file name) > current_epoch_timestamp
        Download zip file

📘

Note

Every Zip file name matches the file's creation timestamp

845