Spacewalk not responding

dolfantimmy · 07-01-2016, 12:15 PM

I have installed Spacewalk following standard documentation. Everything worked, and looks, exactly as it should. However, post installation, I am unable to access the WebUI. Connections always fails with timeout. I have confirmed using netstat that I can reach port 443 on the server. No firewalls are running. I've eliminated any network issues, firewalls, and selinux. Anyone have ideas?

sag47 · 07-02-2016, 12:11 PM

You'll have to describe more about your setup and perhaps a link to the standard documentation you followed.

dolfantimmy · 07-06-2016, 09:22 AM

Centos 7.2.1
Spacewalk 2.3

I used two documents to confirm valid installation:

https://www.unixmen.com/install-and-...k-in-centos-7/
https://fedorahosted.org/spacewalk/wiki/HowToInstall

I CAN telnet to port 443 on the spacewalk server localhost.

I see incoming packets on 443 but no response

Code:

tcpdump -i eno16777728 port 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno16777728, link-type EN10MB (Ethernet), capture size 65535 bytes
10:18:26.126822 IP lvl3adminjmp02.mtips.level3.local.50208 > sp-mcl-vm-000-01.https: Flags [SEW], seq 3931227831, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
10:18:26.126962 IP lvl3adminjmp02.mtips.level3.local.50209 > sp-mcl-vm-000-01.https: Flags [SEW], seq 4130137633, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
10:18:26.387208 IP lvl3adminjmp02.mtips.level3.local.50210 > sp-mcl-vm-000-01.https: Flags [SEW], seq 516584902, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

I see no problems with spacewalk status

Code:

● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2016-07-05 13:44:46 EDT; 20h ago
  Process: 24313 ExecStop=/usr/bin/pg_ctl stop -D ${PGDATA} -s -m fast (code=exited, status=0/SUCCESS)
  Process: 24340 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS)
  Process: 24335 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
 Main PID: 24345 (postgres)
   CGroup: /system.slice/postgresql.service
           ├─ 8968 postgres: rhnuser rhnschema 127.0.0.1(43194) idl
           ├─ 9037 postgres: rhnuser rhnschema 127.0.0.1(43262) idl
           ├─ 9038 postgres: rhnuser rhnschema 127.0.0.1(43264) idl
           ├─ 9158 postgres: rhnuser rhnschema 127.0.0.1(43434) idl
           ├─ 9159 postgres: rhnuser rhnschema 127.0.0.1(43436) idl
           ├─ 9160 postgres: rhnuser rhnschema 127.0.0.1(43438) idl
           ├─ 9161 postgres: rhnuser rhnschema 127.0.0.1(43440) idl
           ├─ 9162 postgres: rhnuser rhnschema 127.0.0.1(43442) idl
           ├─ 9164 postgres: rhnuser rhnschema 127.0.0.1(43444) idl
           ├─ 9165 postgres: rhnuser rhnschema 127.0.0.1(43446) idl
           ├─24345 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
           ├─24346 postgres: logger process                        
           ├─24348 postgres: checkpointer process                  
           ├─24349 postgres: writer process                        
           ├─24350 postgres: wal writer process                    
           ├─24351 postgres: autovacuum launcher process           
           ├─24352 postgres: stats collector process               
           ├─24546 postgres: rhnuser rhnschema [local] idle        
           ├─24721 postgres: rhnuser rhnschema 127.0.0.1(55132) idl
           ├─24722 postgres: rhnuser rhnschema 127.0.0.1(55134) idl
           ├─24723 postgres: rhnuser rhnschema 127.0.0.1(55136) idl
           ├─24725 postgres: rhnuser rhnschema 127.0.0.1(55138) idl
           ├─24726 postgres: rhnuser rhnschema 127.0.0.1(55140) idl
           ├─24728 postgres: rhnuser rhnschema 127.0.0.1(55142) idl
           ├─24729 postgres: rhnuser rhnschema 127.0.0.1(55144) idl
           ├─24730 postgres: rhnuser rhnschema 127.0.0.1(55146) idl
           ├─24731 postgres: rhnuser rhnschema 127.0.0.1(55148) idl
           ├─24732 postgres: rhnuser rhnschema 127.0.0.1(55150) idl
           ├─24733 postgres: rhnuser rhnschema 127.0.0.1(55152) idl
           ├─24734 postgres: rhnuser rhnschema 127.0.0.1(55154) idl
           ├─24735 postgres: rhnuser rhnschema 127.0.0.1(55156) idl
           ├─24736 postgres: rhnuser rhnschema 127.0.0.1(55158) idl
           ├─24737 postgres: rhnuser rhnschema 127.0.0.1(55160) idl
           ├─24756 postgres: rhnuser rhnschema 127.0.0.1(55168) idl
           └─24811 postgres: rhnuser rhnschema 127.0.0.1(55180) idl

Jul 05 13:44:45 sp-mcl-vm-000-01 systemd[1]: Starting PostgreSQL database server...
Jul 05 13:44:46 sp-mcl-vm-000-01 systemd[1]: Started PostgreSQL database server.
● jabberd.service - Jabber Server
   Loaded: loaded (/usr/lib/systemd/system/jabberd.service; enabled; vendor preset: disabled)
   Active: active (exited) since Tue 2016-07-05 13:44:46 EDT; 20h ago
  Process: 24373 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 24373 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/jabberd.service

Jul 05 13:44:46 sp-mcl-vm-000-01 systemd[1]: Starting Jabber Server...
Jul 05 13:44:46 sp-mcl-vm-000-01 systemd[1]: Started Jabber Server.
● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2016-07-05 13:44:46 EDT; 20h ago
  Process: 24248 ExecStop=/usr/libexec/tomcat/server stop (code=exited, status=0/SUCCESS)
 Main PID: 24390 (java)
   CGroup: /system.slice/tomcat.service
           └─24390 /usr/lib/jvm/jre/bin/java -ea -Xms256m -Xmx256m -Djava.awt.headless=true -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024 -XX:MaxNewSize=256 -XX:-UseConcMarkSweepGC -Dnet.sf.ehcache.skipUpdateCheck=true -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Jul 05 13:45:06 sp-mcl-vm-000-01 server[24390]: Jul 05, 2016 1:45:06 PM org.apache.catalina.startup.HostConfig deployDescriptor
Jul 05 13:45:06 sp-mcl-vm-000-01 server[24390]: INFO: Deployment of configuration descriptor /etc/tomcat/Catalina/localhost/rhn.xml has finished in 18,373 ms
Jul 05 13:45:06 sp-mcl-vm-000-01 server[24390]: Jul 05, 2016 1:45:06 PM org.apache.coyote.AbstractProtocol start
Jul 05 13:45:06 sp-mcl-vm-000-01 server[24390]: INFO: Starting ProtocolHandler ["http-bio-127.0.0.1-8080"]
Jul 05 13:45:06 sp-mcl-vm-000-01 server[24390]: Jul 05, 2016 1:45:06 PM org.apache.coyote.AbstractProtocol start
Jul 05 13:45:06 sp-mcl-vm-000-01 server[24390]: INFO: Starting ProtocolHandler ["ajp-bio-127.0.0.1-8009"]
Jul 05 13:45:06 sp-mcl-vm-000-01 server[24390]: Jul 05, 2016 1:45:06 PM org.apache.coyote.AbstractProtocol start
Jul 05 13:45:06 sp-mcl-vm-000-01 server[24390]: INFO: Starting ProtocolHandler ["ajp-bio-0:0:0:0:0:0:0:1-8009"]
Jul 05 13:45:06 sp-mcl-vm-000-01 server[24390]: Jul 05, 2016 1:45:06 PM org.apache.catalina.startup.Catalina start
Jul 05 13:45:06 sp-mcl-vm-000-01 server[24390]: INFO: Server startup in 18620 ms
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2016-07-05 13:45:07 EDT; 20h ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 24228 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 9072 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
 Main PID: 24517 (httpd)
   Status: "Total requests: 2599; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─24517 /usr/sbin/httpd -DFOREGROUND
           ├─24518 /usr/sbin/httpd -DFOREGROUND
           ├─24519 /usr/sbin/httpd -DFOREGROUND
           ├─24520 /usr/sbin/httpd -DFOREGROUND
           ├─24521 /usr/sbin/httpd -DFOREGROUND
           ├─24522 /usr/sbin/httpd -DFOREGROUND
           ├─24523 /usr/sbin/httpd -DFOREGROUND
           ├─24524 /usr/sbin/httpd -DFOREGROUND
           └─24525 /usr/sbin/httpd -DFOREGROUND

Jul 05 13:45:07 sp-mcl-vm-000-01 systemd[1]: Starting The Apache HTTP Server...
Jul 05 13:45:07 sp-mcl-vm-000-01 httpd[24517]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
Jul 05 13:45:07 sp-mcl-vm-000-01 systemd[1]: Started The Apache HTTP Server.
● osa-dispatcher.service - OSA Dispatcher daemon
   Loaded: loaded (/usr/lib/systemd/system/osa-dispatcher.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2016-07-05 13:45:07 EDT; 20h ago
  Process: 24542 ExecStart=/usr/sbin/osa-dispatcher --pid-file /var/run/osa-dispatcher.pid (code=exited, status=0/SUCCESS)
  Process: 24541 ExecStartPre=/bin/rm -f /var/run/osa-dispatcher.pid (code=exited, status=0/SUCCESS)
 Main PID: 24547 (osa-dispatcher)
   CGroup: /system.slice/osa-dispatcher.service
           └─24547 /usr/bin/python -s /usr/sbin/osa-dispatcher --pid-file /var/run/osa-dispatcher.pid

Jul 05 13:45:07 sp-mcl-vm-000-01 systemd[1]: Starting OSA Dispatcher daemon...
Jul 05 13:45:07 sp-mcl-vm-000-01 systemd[1]: PID file /var/run/osa-dispatcher.pid not readable (yet?) after start.
Jul 05 13:45:07 sp-mcl-vm-000-01 systemd[1]: Started OSA Dispatcher daemon.
● rhn-search.service - Spacewalk search engine
   Loaded: loaded (/usr/lib/systemd/system/rhn-search.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2016-07-05 13:45:08 EDT; 20h ago
  Process: 24143 ExecStop=/usr/sbin/rhn-search stop (code=exited, status=0/SUCCESS)
  Process: 24563 ExecStart=/usr/sbin/rhn-search start (code=exited, status=0/SUCCESS)
 Main PID: 24576 (rhnsearchd)
   CGroup: /system.slice/rhn-search.service
           ├─24576 /usr/bin/rhnsearchd /usr/share/rhn/config-defaults/rhn_search_daemon.conf wrapper.pidfile=/var/run/rhn-search.pid wrapper.daemonize=TRUE
           └─24578 /usr/bin/java -Djava.library.path=/usr/lib:/usr/lib64:/usr/lib/oracle/11.2/client/lib:/usr/lib/oracle/11.2/client64/lib:/usr/lib/gcj/postgresql-jdbc:/usr/lib64/gcj/postgresql-jdbc -classpath /usr/share/rhn/search/classes/:/usr/share/rhn/search/lib/commons-lang.jar:/usr/share/rhn/search/lib/commons-logging.jar:/usr/share/rhn/search/lib/mybatis-3.2.3.jar:/usr/share/rhn/search/lib/log4j.jar:/usr/share/rhn/search/lib/lucene-core-2.3.2.jar:/usr/share/java/ojdbc14.jar:/usr/share/rhn/search/lib/picocontainer-1.3.jar:/usr/share/rhn/search/lib/quartz.jar:/usr/share/rhn/search/lib/redstone-xmlrpc.jar:/usr/share/rhn/search/lib/redstone-xmlrpc-client.jar:/usr/share/rhn/search/lib/simple-core.jar:/usr/share/rhn/search/lib/spacewalk-search.jar:/usr/share/java/tanukiwrapper.jar:/usr/share/rhn/search/lib/lucene-analyzers-2.3.2.jar:/usr/share/rhn/search/lib/hadoop-0.18.1-core.jar:/usr/share/nutch:/usr/share/nutch/conf:/usr/share/rhn/search/lib/nutch-2008-12-01_04-01-21.jar:/usr/share/rhn/search/lib/commons-httpclient-3.0.jar:/usr/share/java/slf4j/api.jar:/usr/share/java/slf4j/jcl.jar:/usr/share/java/postgresql-jdbc.jar:/usr/share/java/cglib-nodep.jar:/usr/share/java/objectweb-asm/asm.jar:/usr/share/java/c3p0.jar:/usr/share/java/cglib.jar:/usr/share/java/mchange-commons.jar:/usr/share/java/mchange-commons-java.jar -Dwrapper.key=1FSAPu3jBU1BYlL1 -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=24576 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 com.redhat.satellite.search.Main

Jul 05 13:45:08 sp-mcl-vm-000-01 systemd[1]: Starting Spacewalk search engine...
Jul 05 13:45:08 sp-mcl-vm-000-01 rhn-search[24563]: Starting rhn-search...
Jul 05 13:45:08 sp-mcl-vm-000-01 systemd[1]: Started Spacewalk search engine.
● cobblerd.service - LSB: daemon for libvirt virtualization API
   Loaded: loaded (/etc/rc.d/init.d/cobblerd)
   Active: active (running) since Tue 2016-07-05 13:45:11 EDT; 20h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 24120 ExecStop=/etc/rc.d/init.d/cobblerd stop (code=exited, status=0/SUCCESS)
  Process: 24616 ExecStart=/etc/rc.d/init.d/cobblerd start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/cobblerd.service
           └─24626 /usr/bin/python -s /bin/cobblerd --daemonize

Jul 05 13:45:09 sp-mcl-vm-000-01 systemd[1]: Starting LSB: daemon for libvirt virtualization API...
Jul 05 13:45:11 sp-mcl-vm-000-01 cobblerd[24616]: Starting cobbler daemon: [  OK  ]
Jul 05 13:45:11 sp-mcl-vm-000-01 systemd[1]: Started LSB: daemon for libvirt virtualization API.
RHN Taskomatic is running (24666).

Thanks

dolfantimmy · 07-11-2016, 10:22 AM

If I utilize openssl to attempt to connect to port 443 of the spacewalk server from another host, it failes as follows:

Code:

[root@ln-mcl-vm-000-02 ~]# openssl s_client -connect 10.254.6.60:443
socket: No route to host

However, running the same command from the spacewalk server itself, I get the SSL header back

Code:

openssl s_client -connect localhost:443
CONNECTED(00000003)
depth=0 C = US, ST = VA, O = MyCompany, OU = sp-mcl-vm-000-01, CN = sp-mcl-vm-000-01, emailAddress = root@localhost
(Snippet...  No need to post it all)

So something on the Spacewalk server is not allowing connections from remote machines. As mentioned, SELinux, firewalld, and iptables are either not running or are disabled.

Any ideas?

sag47 · 07-12-2016, 12:10 PM

From the server, is it running on the port you expect?

Code:

netstat -ntlp | grep 443

There may be something in spacewalk that you need to configure to listen on the public interfaces. Check your ethernet device names vs settings and how it's listening.

Are the two machines you're trying to connect in the same subnet address space? If not, how are you routing traffic for different subnets? The no route to host seems more likely

dolfantimmy · 07-13-2016, 09:00 AM

Thanks Harry. I have already confirmed that the spacewalk server is listening on port 443. Both the source and destination hosts are on the same subnet.

I thought about your suggestion previously regarding public interfaces. Not sure where this would be configured however.

sag47 · 07-13-2016, 11:43 AM

What I meant about public inerfaces was anything "not localhost". i.e. the interface incoming traffic would traverse to get to the spacewalk service.

No route to host also indicates iptables. Do you have inbound iptables rules on the server? Do you have outbound iptables rules on the client trying to connect?

Another possibility. Do you have SELinux enabled? If so, it may not be optimally configured for the spacewalk functionality you wish to use (or some component of it). Try troubleshooting it. If you've not troubleshoot SELinux before then refer to some blog posts that I've made troubleshooting other applications. http://www.linuxquestions.org/questi...ux-adventures/