How to secure a refresh token?

Question

I'm using JWTs for authenticating users for my app. When a user logs in they are given an access token and a refresh token. To keep the refresh token safe, I don't store it on the client-side, but save it on the back-end with their account so it's not easy to access. I'm confused about the security of refresh tokens though, here's the logic that I'm understanding when I read online resources on how to use refresh tokens:

1. authenticate
2. store access token + refresh token somewhere (in my case, access token on the front-end and refresh token on the back-end)
3. when performing an api request, validate the access token on the api side
4. if the access token is expired, use the refresh token to generate a new access token + new refresh token, send access token back to client
5. store tokens as before... and repeat

The security issue I'm worried about is if someone else (hacker) got a hold of the access token and they send a request to the api with it, if the token is expired the api will use the refresh token to get a new access token + new refresh token and return at least the access token to the hacker.

I read this article about 5-6 times and I read this article a few times, as well as some other articles on the subject, they all say something along the lines of

make sure to store the refresh token securely because it's long lived, the access_token is short lived so not as big of a deal

But according to the flow I described above, it doesn't matter if the access token is short lived, the refresh token will be used to get a new access token and have access forever.

Is there something I'm missing? How would the api know who is sending the request if a hacker got a hold of the expired access token? it will still send a new one using the refresh token. Am I supposed to somehow validate who is sending the request?

---

UPDATE

So I do understand that when a new access token is requested, I need to send over the refresh token, the client ID, and the client secret. The issue I have with that is, like before, the hacker can send a request to my API server, the server gets the hijacked access token from the hacker, it will see that it's expired, so it will send the refresh token, along with the clientID/client secret (which are stored as environment variables) to the Auth API and get back a new access token / refresh token, which brings us back to the same issue.

---

UPDATE 2

some interesting questions on the subject:

1. Why Does OAuth v2 Have Both Access and Refresh Tokens?
2. https://security.stackexchange.com/questions/87119/how-secure-are-expiring-tokens-and-refresh-tokens

according to the second question and answer, it seems like the refresh token is not a more secure way to maintain access, it's just that it's easier to detect a hacker because auth/refresh tokens keep getting requested and invalidating the other's tokens. The issue with this is this will only happen if 2 users are simultaneously trying to access resources - if only the hacker happens to be active at a given time period, he will have unlimited access to the original users data until the original user tries to use the app and access protected resources

Answer 1

TL;DR (an option for improving user experience)

1. Store a short lived access token in-memory OR localstorage.
2. Store refresh token in-memory (only)

now as long as the user doesnt do a hard refresh or navigate away from your site, they can be logged in forever.

---

General thoughts

I'm not an expert on all the intricacies of the different tokens, their mechanisms, and their storage best practices (so refer to other articles/experts on the subject - Tim Hardy poses an excellent and strong counterargument to my findings in the comments below) but it seems like a bad idea to use refresh tokens with browser based apps. Refresh tokens can be stored securely on phones/other devices. You could potentially use an http only cookie in a browser or store the token in memory (see more below) but again, I'm not certain on the security of this approach (I'm not saying its NOT secure, I'm saying I don't know how secure)

---

Improving user experience

while (again) I personally dont know the full extent of risks involved with storing access/refresh tokens in browser storage (besides XSS attacks), you have a couple relatively safe options here.

Either store a long-lived access token ONLY in-memory, or use a combination of a short-lived access token (in-memory or localstorage is safe too probably since its short-lived) plus a refresh token ONLY in-memory. This approach allows the user to be logged in forever as long as they dont refresh the page or navigate away.