You are not logged in.

#1 2009-08-26 02:50:42

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

[SOLVED] cURL, PHP, Apache, Insanity!

Hello,

I'm about to throw my server out the window. Thought I'd ask here before I do though. wink

tl;dr: cURL doesn't work when it's used in a PHP application and running through a web browser. However, that same PHP page with cURL, when run via the terminal, does what it's supposed to do and fetches the information that I want. wtf?

I'm having networking problems, but I don't know why. I'm trying to create a web application in PHP (served by Apache) that will fetch an XML page via cURL. However, when I run it, I get errors. I've been testing it over the last few days, and have been able to reproduce via a few methods and make it work via other methods. Here are my tests:

Like I said, I've created something that will pull XML data from a website and save it to the server. The script is multiple pages and Class-based (as opposed to procedural based). When running the script via a web-browser and get's to the part where it executes the curl function, I get this (via a var_dump()):

"Couldn't resolve host 'www.websitehere.com'"

So I made a test page that didn't have any fancy OOP or included pages; one that should work, since it's an example given at the PHP website itself. The result when running it from a browser?

"Couldn't resolve host 'www.example.com'"

So then I thought 'Maybe it's curl itself. To test this, I opened up a SSH session and ran this command: curl example.com I got this as a result:

<HTML>
<HEAD>
  <TITLE>Example Web Page</TITLE>
</HEAD>
<body>
<p>You have reached this web page by typing "example.com",
"example.net",
  or "example.org" into your web browser.</p>
<p>These domain names are reserved for use in documentation and are not available
  for registration. See <a href="http://www.rfc-editor.org/rfc/rfc2606.txt">RFC
  2606</a>, Section 3.</p>
</BODY>
</HTML>

So it's obviously not cURL itself. It must be a PHP problem then, right? After playing around with it a few more times, I decided to run the PHP example linked above from the terminal instead of the web browser: /usr/bin/php ./example.php

Viola, it works! So I run my script using the same technique: /usr/bin/php ./script.php And what do you know? It works also! cURL successfully connects and retrives the XML data, and it also save it to the directory I wanted it to be saved in.

So now I ask: Why is this? Why does cURL return Couldn't resolve host when the PHP script is run in a web browser, but is fine when run in the terminal? Is this a PHP problem? An Apache problem? Below, I've included some network information that may help:

My router (running dd-wrt) has the option to set a domain name. I've never really understood what this does, but have always set it to my last name + .net (holmes.net) although that isn't register online or anything. Whenever I set up a computer and it asked me to input a domain name, that's what I always put in. The server (hostname `server`) has a Static DHCP IP assigned to it via it's MAC address; it is always 192.168.1.10. The router is 192.168.1.1. Here's some network files from the server:

#
# /etc/host.conf
#

order hosts,bind
multi on
#
# /etc/hosts: static lookup table for host names
#

#<ip-address>   <hostname.domain.org>   <hostname>
127.0.0.1               localhost.localdomain   localhost server

# End of file
# Generated by dhcpcd from eth0
# /etc/resolv.conf.head can replace this line
domain holmes.net
nameserver 192.168.1.1
# /etc/resolv.conf.tail can replace this line

And that should be it. If anyone needs more info, I'll be happy to supply it. I have a feeling that it's the domain settings (the holmes.net crap) that is screwing with this. Maybe it's a setting in Apache that I forgot to set that's causing the problem. Or maybe it's a bug in PHP (I've read that it had issues back in the early 2000's with cURL and the resolving error). I dunno what it is.

Can anyone help?

Last edited by XtrmGmr99 (2009-11-03 04:19:13)

Offline

#2 2009-08-26 03:35:33

neddie_seagoon
Member
Registered: 2009-08-23
Posts: 121

Re: [SOLVED] cURL, PHP, Apache, Insanity!

It's odd that the same script run from the command line works fine, but maybe check this link and try putting up a test script as poster chanh describes:
http://www.000webhost.com/forum/custome … -curl.html

Also, you can try enabling fopen for URLs, and trying to open or include some remote file - does it fail resolving the hostname as well? If it does then the issue is with the PHP running through Apache.

Also, maybe try switching to using PHP as a CGI module instead of Apache module, to see if that makes any difference.

Offline

#3 2009-08-26 04:06:24

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

neddie_seagoon wrote:

It's odd that the same script run from the command line works fine, but maybe check this link and try putting up a test script as poster chanh describes:
http://www.000webhost.com/forum/custome … -curl.html

Also, you can try enabling fopen for URLs, and trying to open or include some remote file - does it fail resolving the hostname as well? If it does then the issue is with the PHP running through Apache.

Also, maybe try switching to using PHP as a CGI module instead of Apache module, to see if that makes any difference.

What do you mean put up a test script?

I've tried getting a file from fopen. It also fails:

Warning: fopen() [function.fopen]: php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution in /home/ryan/public_html/xgamer.ath.cx/test/test.php on line 3

Warning: fopen(http://www.example.com/) [function.fopen]: failed to open stream: php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution in /home/ryan/public_html/xgamer.ath.cx/test/test.php on line 3
bool(false)

As to your last suggestion... I have no idea what you just said. tongue CGI? What?

Oh, and I found this. It's from Google's cache, becasue the main site is down, but it sounds like the exact same problem. He says he fixed it by changing an Apache setting; problem is, he doesn't mention which setting... <_<

Offline

#4 2009-08-26 04:53:14

neddie_seagoon
Member
Registered: 2009-08-23
Posts: 121

Re: [SOLVED] cURL, PHP, Apache, Insanity!

Can you try a couple things:
Edit your /etc/resolv.conf and comment out the existing lines (incl. the domain one) and add:
nameserver 208.67.222.222
nameserver 208.67.220.220

(these are OpenDNS nameservers - http://wiki.archlinux.org/index.php/OpenDNS)

Then restart Apache and see if your scripts can resolve names.

What do you mean put up a test script?

I just meant to create a new test script (following that link's example) on your server and browse to it.

CGI? What?

smile I don't recall the exact steps to switch Apache from using mod_php to php-cgi, but a quick search should tell you the httpd.conf modifications to do this. Also if you are running multiple live sites on this box you would want to be sure to only do it for the specific virtualhost. But first try the stuff above and see how it works out.

Offline

#5 2009-08-26 05:04:00

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

neddie_seagoon wrote:

Can you try a couple things:
Edit your /etc/resolv.conf and comment out the existing lines (incl. the domain one) and add:
nameserver 208.67.222.222
nameserver 208.67.220.220

(these are OpenDNS nameservers - http://wiki.archlinux.org/index.php/OpenDNS)

Then restart Apache and see if your scripts can resolve names.

What do you mean put up a test script?

I just meant to create a new test script (following that link's example) on your server and browse to it.

CGI? What?

smile I don't recall the exact steps to switch Apache from using mod_php to php-cgi, but a quick search should tell you the httpd.conf modifications to do this. Also if you are running multiple live sites on this box you would want to be sure to only do it for the specific virtualhost. But first try the stuff above and see how it works out.

About the test script - I've already created a few test scripts. smile There's one fore cURL, one for fopen, and my custom script that making.

I switched the name servers and restarted Apche. Didn't work.

Normally, I have my router pointing to the OpenDNS servers, and my server pointing to my router. Besides, it wouldn't be a DNS problem - I can ping any site just fine. It's an internal problem on either the server or network. smile

About the CGI thing -- I'll look into it tomorrow. I really don't know what CGI is exactly, but it shouldn't be too hard to switch, like you said. smile

EDIT: Now I can't seem to get the server connected to the router. I've put in the old nameserver info, but I can't connect... >.<

EDIT: All is fine with my server's connection once more. smile

Last edited by XtrmGmr99 (2009-08-27 07:27:02)

Offline

#6 2009-08-27 02:17:56

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

Well, I've searched and still can't really find an answer to this... It's driving me nuts! I'm gonna try the CGI switch...

EDIT: and alas! I can't find info on how to go to CGI... probably because I don't know what I'm supposed to be searching for. tongue

Last edited by XtrmGmr99 (2009-08-27 02:43:09)

Offline

#7 2009-08-27 02:53:35

neddie_seagoon
Member
Registered: 2009-08-23
Posts: 121

Re: [SOLVED] cURL, PHP, Apache, Insanity!

Check this article:
http://www.gentoo.org/proj/en/php/php4- … ration.xml

You would need to adapt the instructions there roughly like this (I think):

Move the php config file out of /etc/httpd/conf.d (if that is where it is) so Apache doesn't automatically load the module.

Add the following to httpd.conf:
 

  ScriptAlias /php5-cgi /path/to/php-cgi    
    Action php5-cgi /php5-cgi
    AddHandler php5-cgi .php

Restart Apache, and your PHP files should be handled by the CGI version of PHP -- phpinfo() should tell you that it's not using the Apache 2 handler, but rather CGI/FastCGI.

I haven't tested this at all, but it should work. wink

Offline

#8 2009-08-27 03:07:43

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

i'll try that, but I'm not too sure if that's gonna help at all. >_>

Offline

#9 2009-08-27 04:18:24

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

BREAKTHROUGH!!! Okay, admittedly, I didn't try the CGI thing. Looked too complicated... But I found this. They suggested that it's a permissions problem with the Apache user.

So I tried changing the Apache user from http to root, then restarted Apache. It gave `Unable to Connect` server errors. I then, however, switched Apache back to the http user, restarted Apache, and OMG! it works! The cURL, the fopen, everything from what I can tell!

The catch: it only works until the server is restarted. Once the server is restarted, it goes back to the way it was: I'd have to switch the Apache user to root (in which no pages are able to load) and then switch back to the http user in order for cURL and any other 'website fetching function' to work...

So I've come to the conclusion that it's a permission problem with the http user. My next question would be what do I do now? I'm really stumped now, but this feels soo close to be solved.

Offline

#10 2009-08-27 04:39:26

neddie_seagoon
Member
Registered: 2009-08-23
Posts: 121

Re: [SOLVED] cURL, PHP, Apache, Insanity!

Wow, that is extremely odd! hmm

I've never come across this before so you could try some other things:
-switch to some other user like nobody or create a brand new one to test
-try giving the user a shell
-make sure permissions are right (755) on any files apache needs to access
-does it still work if you fully stop and start apache with the user http set?
-try the commandline test with su http -c php test.php (not sure of exact syntax)

These are all kinda shot in the dark suggestions but I am very curious to know what solves this!

Offline

#11 2009-08-27 04:55:27

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

neddie_seagoon wrote:

Wow, that is extremely odd! hmm

I've never come across this before so you could try some other things:
-switch to some other user like nobody or create a brand new one to test
-try giving the user a shell
-make sure permissions are right (755) on any files apache needs to access
-does it still work if you fully stop and start apache with the user http set?
-try the commandline test with su http -c php test.php (not sure of exact syntax)

These are all kinda shot in the dark suggestions but I am very curious to know what solves this!

1) Using a user like `nobody` spits back the same stuff as `http` does. Same thing as creating a new user...

2) Give the http user a shell? Doing that doesn't do anything...

3) Which files would those be? That's what i was kinda wondering: what program files does the Apache user need to execute (ex: the cURL program?) in order to not give any errors out? Something that only root can read (also ran Apache as my personal user and still didn't work, so it has nothing to do with my web files which are 777 anyway)

4) You mean like `/etc/rc.d/httpd stop` and then `/etc/rc.d/httpd start`? I've been using the restart command, which, theoretically, does the exact same thing (but just to be sure: no, specifically starting and stopping Apache with the `http` user has no apparent effect)

5) As soon as I try to log into the http user, Arch give this:

# su http -c php test.php
You are required to change your password immediately (root enforced)
Changing password for http.
(current) UNIX password:

I don't know the password, thus cannot proceed. If tried a blank password, `password`, and `http` as the passwords, with no go.

It must be permissions. there has to be something that root has access to but the http user doesn't... hm... =/

Last edited by XtrmGmr99 (2009-08-27 04:57:42)

Offline

#12 2009-08-28 03:14:29

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

Shameless bump. smile

I've continued to search Google with keyworks involving Apache and program permissions... Still haven't come up with anything...

Offline

#13 2009-08-28 03:37:02

neddie_seagoon
Member
Registered: 2009-08-23
Posts: 121

Re: [SOLVED] cURL, PHP, Apache, Insanity!

You could try to check /etc/httpd and follow any symlinks there and set things to 755. And all the PHP binaries/libs. Also, do you see any errors in your Apache error log when the setup is not working? Check /var/log/httpd/error.log or something like that, maybe it will point you in the right direction.

Offline

#14 2009-08-28 03:42:08

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

neddie_seagoon wrote:

Wow, that is extremely odd! hmm
-does it still work if you fully stop and start apache with the user http set?

Actually, after reading this and leaning that Apache stored various things in a cache I just tried this again. It works! I dunno why explicitly giving the `start` and `stop` commands didn't work last time (could be because of effects that had to do with other experiments at trying to solve the problem?), but they worked this time...

However, this suffers from the same situation the other working method suffered from: it works until the server restarts. After that, you must manually stop and start the Apacher server again for the cURL function to work. I'll be looking into the start-ups scrips and see if I can find something useful. As a last resort, I'll just give the manual stop and start commands in rc.local to be run after everything else boots.

Will report back with anything I find. Maybe this thread can be helpful to someone else that googles the same problem smile

Offline

#15 2009-08-28 04:00:29

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

I think I found the problem:

My old DAEMONS line used to look like this:

DAEMONS=(syslog-ng @network @sshd @inadyn mysqld httpd @netfs @samba crond)

The reason why I had the network start in the background was because many months ago when I first installed Arch I ran into network problems which would hang the computer for about 60 seconds. So I put it in the background. It's fine now, I just never removed the background flag.

Apparently, network pulls in DNS info that Apache then reads only once when it starts up. If Apache is started before the network can finish initializing then it pulls non-existent DNS data, thus producing errors. This must be why cURL and pings worked from the terminal - they must access DNS info on-the-fly, instead of from a cache like Apache does. Forcing network to start before Apache successfully allows Apache to run PHP pages w/ cURL and fopen commands.

Thank you very much neddie for helping me throughout this. I very much appreciate it. smile

Offline

#16 2009-08-28 04:20:47

neddie_seagoon
Member
Registered: 2009-08-23
Posts: 121

Re: [SOLVED] cURL, PHP, Apache, Insanity!

Excellent to hear you've tracked down the issue! I hope I didn't send you on too many wild-goose chases.. wink

I'm sure this bit of information will come in handy if I ever see a similar issue. On a side note, while it's good you didn't try the PHP CGI thing (so as to not needlessly complicate your Apache config), I have a feeling it would have worked, because:
-right now your PHP is handled by an Apache module, which would get its DNS information from Apache
-switching to CGI would run the php-cgi binary for each PHP script (like running it on the command-line), so it would pull in its own DNS info each time (I think).
But regardless, if you had switched to that it may have worked but the root issue would have still been there. So good thing you didn't! smile

Offline

#17 2009-08-28 05:07:07

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

neddie_seagoon wrote:

Excellent to hear you've tracked down the issue! I hope I didn't send you on too many wild-goose chases.. wink

I'm sure this bit of information will come in handy if I ever see a similar issue. On a side note, while it's good you didn't try the PHP CGI thing (so as to not needlessly complicate your Apache config), I have a feeling it would have worked, because:
-right now your PHP is handled by an Apache module, which would get its DNS information from Apache
-switching to CGI would run the php-cgi binary for each PHP script (like running it on the command-line), so it would pull in its own DNS info each time (I think).
But regardless, if you had switched to that it may have worked but the root issue would have still been there. So good thing you didn't! smile

Ah, I love wild-goose chases. That's the way you learn and better yourself (and your configs too wink)

big_smile

Offline

#18 2009-11-02 21:42:36

rudle
Member
Registered: 2009-11-02
Posts: 1

Re: [SOLVED] cURL, PHP, Apache, Insanity!

I was having the exact same issue as described here and the '/etc/rc.d/httpd stop ; /etc/rc.d/httpd start' tip did the trick for me.  I have a similar DAEMONS array, so that is likely the issue.

Thanks!

Offline

#19 2009-11-03 04:20:30

XtrmGmr99
Member
Registered: 2009-04-14
Posts: 128

Re: [SOLVED] cURL, PHP, Apache, Insanity!

rudle wrote:

I was having the exact same issue as described here and the '/etc/rc.d/httpd stop ; /etc/rc.d/httpd start' tip did the trick for me.  I have a similar DAEMONS array, so that is likely the issue.

Thanks!

I'm glad this topic helped you out. I've been running a problem free Apache ever since I fixed the DAEMONS array. Hope it works out for you big_smile

Offline

#20 2011-01-25 15:56:24

hughitt1
Member
Registered: 2010-02-18
Posts: 77
Website

Re: [SOLVED] cURL, PHP, Apache, Insanity!

Just a quick clarification:

"/etc/rc.d/httpd restart" won't work: you must actually tell Apache to "stop" and then "start" again.

Also, if you are using NetworkManager, you can still run into this problem even if httpd comes after NetworkManager. For example, I had:

DAEMONS=(syslog-ng dbus networkmanager httpd mysqld netfs crond alsa gdm)

and I still have problems after rebooting. I'm going to try moving httpd further back in the ordering and see if that resolves things. Otherwise I may try backgrounding it.

Best,
Keith

Offline

#21 2011-02-03 19:24:06

Dave Cohen
Member
Registered: 2008-09-27
Posts: 126

Re: [SOLVED] cURL, PHP, Apache, Insanity!

Had the same problem.  stopping and restarting apache fixed it, for the time being.

But I'm already starting network daemon in the foreground.  There must be some other cause that can corrupt apache's DNS cache.

At any rate I'm glad I found this thread.  Thanks all.

Offline

Board footer

Powered by FluxBB