Fetching data from Twitter

I started sending requests to the Twitter API for my liked tweets. I did some testing in Postman first.

The Twitter API's liked tweets endpoint returns one page at a time. One page has around 100 liked tweets on it. Pages are connected by pagination tokens. Each page of tweets returned from the endpoint has two pagination tokens on it. One that points to the previous page and one that points to the next page. To get from one page to another, you must use the pagination token you get from one page and make a new request with a query parameter containing that token. Your new request will return the next page.

To fetch as many tweets as I could at a time, I wrote a loop that fetched a page, then used the pagination token from that page to fetch the next page ad infinitum. Unfortunately, Twitter rate-limits you to 75 requests per 15-minute period, so I had to make my requests, wait, and then go for it again after the cooldown. I have almost 20k liked tweets. This took a while.

This is what the full HTTP GET request looks like. Straight from the Postman console. I used the bearer token from one of my projects on my free tier Twitter Developer account to authenticate myself. That's the header that says "Authorization: Bearer AAAAAAAAAA███████████████████████████████████████████████████████████████████████████████."

Not gonna share the actual bearer token with you, obviously.

After extracting all that data, it was time to transform it.