David BlumeThis is for @nylabs on Twitter, regarding this thread: https://twitter.com/dblume/status/588942939771961344
I love the OpenPaths project, and have been successfully using it for years. Thank you for making it! Regarding your request to send you my API call, via Twitter? Twitter's not really good for that. Here's what I use:
https://openpaths.cc/api/1?oauth_body_hash=[hash]&oauth_nonce=[nonce]&oauth_timestamp=[timestamp]&start_time=[starttime]&oauth_signature_method=HMAC- SHA1&oauth_version=1.0&end_time=[endtime]&oauth_signature=[mysig]&oauth_consumer_key=[mykey]
It's built like https://openpaths.cc/api (Except YOURACCESSKEY and YOURSECRETKEY are populated correctly, of course.)
When the server doesn't emit 502s and 504s, it returns valid data. So the problem doesn't seem to be the API call.
The problem appears to be when my cronjob makes the API call. My timezone is US/Pacific, so keep that in mind for this next part: Look at the clues in the logs at: https://david.dlma.com/location/stats.txt
The evening calls have been getting slower and slower over the past few months, until 2015-03-08, when the evening calls have been returning 502s and 504s once the 60s barrier was broken.
Here's an interesting clue: When I try to manually debug it, the first call might sometimes take a long time, but the subsequent ones are often back in the 1-4s range. (Notice the three calls I made after 2015-04-17, 21:47.) It's almost as if it takes a call to "load" or "wake up" the service.
You wrote, "We’ve checked and nothing seems to be blocking or getting delayed."
I've even tried to "spin up" or "prime" your server by sending some valid requests before the request that I care about. But that didn't fix the issue at 21:36. (Is 00:36 US/Eastern a meaningful time for your servers? Do they do some maintenance that'd eventually take longer and longer?)
Anyway, here's what I'm going to do on my side: Try to isolate the problem on the client (my) side, or the server (your) side. I already make other API calls to Amazon, Google, Twitter, Instagram, Plurk and more. Ex., http://stats.dlma.com/ They all usually work fine, all day long.
To better support the theory, I'll have the same cronjob that invokes your API endpoint also invoke others. If the others continue to be fine, but only openpaths.cc/api/1 is the only one emitting 502s and 504s, then that'll suggest the problem is on your server. If the other API calls also are slow to respond, it'll point the finger at my client.
We'll have to wait for a few more 21:36 PSTs to pass to gather the results.
I love the OpenPaths project, and have been successfully using it for years. Thank you for making it! Regarding your request to send you my API call, via Twitter? Twitter's not really good for that. Here's what I use:
https://openpaths.cc/api/1?oauth_body_hash=[hash]&oauth_nonce=[nonce]&oauth_timestamp=[timestamp]&start_time=[starttime]&oauth_signature_method=HMAC- SHA1&oauth_version=1.0&end_time=[endtime]&oauth_signature=[mysig]&oauth_consumer_key=[mykey]
It's built like https://openpaths.cc/api (Except YOURACCESSKEY and YOURSECRETKEY are populated correctly, of course.)
When the server doesn't emit 502s and 504s, it returns valid data. So the problem doesn't seem to be the API call.
The problem appears to be when my cronjob makes the API call. My timezone is US/Pacific, so keep that in mind for this next part: Look at the clues in the logs at: https://david.dlma.com/location/stats.txt
The evening calls have been getting slower and slower over the past few months, until 2015-03-08, when the evening calls have been returning 502s and 504s once the 60s barrier was broken.
Here's an interesting clue: When I try to manually debug it, the first call might sometimes take a long time, but the subsequent ones are often back in the 1-4s range. (Notice the three calls I made after 2015-04-17, 21:47.) It's almost as if it takes a call to "load" or "wake up" the service.
You wrote, "We’ve checked and nothing seems to be blocking or getting delayed."
I've even tried to "spin up" or "prime" your server by sending some valid requests before the request that I care about. But that didn't fix the issue at 21:36. (Is 00:36 US/Eastern a meaningful time for your servers? Do they do some maintenance that'd eventually take longer and longer?)
Anyway, here's what I'm going to do on my side: Try to isolate the problem on the client (my) side, or the server (your) side. I already make other API calls to Amazon, Google, Twitter, Instagram, Plurk and more. Ex., http://stats.dlma.com/ They all usually work fine, all day long.
To better support the theory, I'll have the same cronjob that invokes your API endpoint also invoke others. If the others continue to be fine, but only openpaths.cc/api/1 is the only one emitting 502s and 504s, then that'll suggest the problem is on your server. If the other API calls also are slow to respond, it'll point the finger at my client.
We'll have to wait for a few more 21:36 PSTs to pass to gather the results.
Comments
My use case? Just to keep track of where I've been and where I'm likely to be going. See the "Secret Mode" part at the bottom of this post: http://david.dlma.com/blog/my-location-predictor
The openpaths.cc website came back by 9:44pm US/Pacific. The API started returning 200s, but with no data! Then at 9:47pm, it started returning real data. (Now look at the production job here: http://david.dlma.com/location/stats.txt) Look at the calls made on 2015-04-20. Those "0s OK" are indicating no points were returned. Then at 21:47, the evening's points were returned.
I think we have enough evidence that you should give it a look. Maybe an internal server has too many logs or temp files clogging it up? It's doing a maintenance job that should be short but is now taking minutes?
Good luck. Let me know if there's anything I can do to help.