Cleaning a Web SiteThis relatively short post shows how I installed and ran the W3C LogValidator on my remotely hosted web site. It is another one of those posts on a very narrow topic which I suspect will be of interest to only a few. Like many others, the web hosting service provider offers its clients the cPanel control software to mangage their web site. The provider also allow me to open SSH sessions into my account. I ended up using these two methods concurrently to install and run the W3C validator.
First thing first, I wanted to verify that perl was available and that the available version was recent enough before trying to install anything. This was easily done in an SSH session.
[sigmdelc@peel ~]$ which perl
/usr/bin/perl
[sigmdelc@peel ~]$ perl -v
This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi
(with 44 registered patches, see perl -V for more detail)
...
Almost everything was obtained from the Comprehensive Perl Archive Network (CPAN) using the interactive tool Perl Modules in the SOFTWARE section of cPanel.
As per Dependencies : what Log Validator needs in How to install the Log Validator the following modules must be installed.
- LWP::UserAgent
- Sys::Hostname::Long
- Config::General
- File::Temp
- DB_File
- Mail::Sendmail
- URI::Escape
- WebService::Validator::CSS::W3C
Finally, the main script
- W3C::LogValidator
can be installed in the same manner. Simply enter each module's name in the Install a Perl Module text box and press the button as shown below.
Let's check that the validator has been installed.
[sigmdelc@peel ~]$ cd perl5
[sigmdelc@peel perl5]$ ls bin/logprocess.pl
logprocess.pl
Two things caused problems which made it impossible to run the validator. First off, the module could not find anything.
[sigmdelc@peel bin]$ ./logprocess.pl -h
Can't locate W3C/LogValidator.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./logprocess.pl line 13.
BEGIN failed--compilation aborted at ./logprocess.pl line 13.
However, we have already seen the solution for that. Changing the first line in the script, #!/usr/bin/perl -w to #!/usr/bin/perlml does work. When I first tried to do that with nano, it was not possible to save the modified file. File attributes turned out to be the problem.
[sigmdelc@peel bin]$ ls -l logprocess.pl
-r-xr-xr-x 1 sigmdelc sigmdelc 2266 Nov 20 11:04 logprocess.pl
no one has write privileges!
[sigmdelc@peel bin]$ chmod a+w logprocess.pl
[sigmdelc@peel bin]$ ls -l logprocess.pl
-rwxrwxrwx 1 sigmdelc sigmdelc 2266 Nov 20 11:04 logprocess.pl
With that change, it was possible to change the first "shebang" line of the script which after that could start.
[sigmdelc@peel bin]$ ./logprocess.pl -h
defined(@array) is deprecated at /home/sigmdelc/perl5/lib/perl5/W3C/LogValidator.pm line 563.
(Maybe you should just omit the defined()?)
Usage: logprocess.pl [options]
General options are:
-h/--help help. what you're reading now.
-f/--config <filename> to read a specific config file. strongly suggested.
Verbosity options are (default - no option - is between quiet and verbose):
-q/--quiet quiet. not verbose. useful for cron'd usage.
-v/--verbose verbose. lots of blah blah.
-d/--debug even more verbose than verbose, for debug purpose.
Output choices are (default - nothing specified - is command-line output):
--HTML trigger output in HTML
--email trigger output as mail sent to maintainer
-o/--output <filename> choose where to save output
the output will go to console if not specified
-s/--sendto <email> choose where to send the e-mail
In e-mail output mode, if neither this nor ServerAdmin (in the config)
is specified then the output will fall-back to console
Note:
All these options can be configured through the config file.
Setting an option overrides the choice made in the config file.
The script is not able to do much because the configuration file was not installed or if it was, neither logprocess.pl nor I could find it.
[sigmdelc@peel perl5]$ find . -name logprocess.conf
[sigmdelc@peel perl5]$ find . -name *conf
[sigmdelc@peel perl5]$
A sample configuration is available, see Where is the (sample) configuration file? in How to use the Log Validator if the previous link does not work. I created a config directory and used nano to create the file in that directory by copying and pasting the content of the sample. Ugly and impossible to show here, but it works.
[sigmdelc@peel perl5]$ mkdir config
[sigmdelc@peel perl5]$ cd config
[sigmdelc@peel config]$ nano logprocess.conf
copy, paste and save!
[sigmdelc@peel config]$ls
logprocess.conf
OK, but the default values definitely did not work.
[sigmdelc@peel perl5]$ bin/logprocess.pl -f config/logprocess.conf
defined(@array) is deprecated at /home/sigmdelc/perl5/lib/perl5/W3C/LogValidator.pm line 563.
(Maybe you should just omit the defined()?)
could not open log file /var/log/apache/access.log : No such file or directory at /home/sigmdelc/perl5/lib/perl5/W3C/LogValidator.pm line 317.
Reading logfiles: [sigmdelc@peel perl5]$
Here are the changes I made to the configuration file. I think the change to line 28 is not strictly needed but those to lines 49, 55 and 63 (the numbers are approxiamete) are necessary.
| Line | Original | New content |
|---|---|---|
| 28 | ServerAdmin me@my.domain | michel@sigmdel.ca |
| 49 | DocumentRoot /var/www/ | DocumentRoot /home/sigmdelc/public_html |
| 55 | ServerName my.web.site | ServerName sigmdel.ca |
| 63 | CustomLog /var/log/apache/access.log combined | CustomLog /home/sigmdelc/logs/sigmdel.ca-ssl_log-Oct-2021 combined |
That last change required finding the log files and, since these were compressed to extract at least one.
[sigmdelc@peel ~]$ cd logs
[sigmdelc@peel logs]$ gz -d sigmdel.ca-ssl_log-Oct-2021.gz
Moving back to the perl5 directory, I launched the W3C log validator and it worked.
[sigmdelc@peel logs]$ cd ..
[sigmdelc@peel ~]$ cd perl5
[sigmdelc@peel perl5]$ bin/logprocess.pl -f config/logprocess.conf
defined(@array) is deprecated at /home/sigmdelc/perl5/lib/perl5/W3C/LogValidator.pm line 563.
(Maybe you should just omit the defined()?)
Reading logfiles: Done!
Now Using the Basic module...
Done!
Now using the HTML Validator module... processing 1780 URIs
(This may take a long time if you have many files to validate)
trimmed list to 295 URIs
Done!
Now Using the CSS Validation module...
Done!
************************************************************************
Results for module basic
************************************************************************
Here are the 100 most popular documents overall for sigmdel.ca.
Rank Hits Address
------ ------ -----------------------------------------------------------------------
1 4429 http://sigmdel.ca/michel/md3262.png
2 4290 http://sigmdel.ca/michel/to.png
3 3089 http://sigmdel.ca/michel/css/base2.css
4 3089 http://sigmdel.ca/michel/home-icon.png
5 3088 http://sigmdel.ca/michel/about-icon.png
6 3087 http://sigmdel.ca/michel/downloads-icon.png
7 3086 http://sigmdel.ca/michel/archives-icon.png
8 2889 http://sigmdel.ca/michel/arrow-left.png
9 2805 http://sigmdel.ca/michel/arrow-up.png
10 2512 http://sigmdel.ca/michel/arrow-right.png
11 1531 http://sigmdel.ca/michel/fr-icon.png
12 1415 http://sigmdel.ca/michel/css/base.css
13 1404 http://sigmdel.ca/michel/loupe17.png
14 860 http://sigmdel.ca/michel/ha/wireguard/wireguard_02_en.html
15 767 http://sigmdel.ca/michel/ha/wireguard/img/public_ip_router.jpg
16 745 http://sigmdel.ca/michel/ha/wireguard/img/android_wg_06.jpg
17 743 http://sigmdel.ca/michel/ha/wireguard/img/router_port_forwarding.jpg
18 737 http://sigmdel.ca/michel/ha/wireguard/img/zenfone_1.jpg
19 737 http://sigmdel.ca/michel/ha/wireguard/img/android_wg_01.png
20 737 http://sigmdel.ca/michel/ha/wireguard/img/android_wg_02.png
21 735 http://sigmdel.ca/michel/ha/wireguard/img/configs.png
22 734 http://sigmdel.ca/michel/ha/wireguard/img/android_wg_04.png
23 732 http://sigmdel.ca/michel/ha/wireguard/img/android_wg_03.png
24 729 http://sigmdel.ca/michel/ha/wireguard/img/zenfone_2.jpg
25 724 http://sigmdel.ca/michel/ha/wireguard/img/win10_client_02.jpg
26 722 http://sigmdel.ca/michel/ha/wireguard/img/win10_client_03.jpg
27 721 http://sigmdel.ca/michel/ha/wireguard/img/win10_client_01.jpg
28 719 http://sigmdel.ca/michel/ha/wireguard/img/win10_client_04.jpg
29 718 http://sigmdel.ca/michel/ha/wireguard/img/win10_PSFTP.jpg
30 715 http://sigmdel.ca/michel/ha/wireguard/img/win10_client_icon.jpg
31 630 http://sigmdel.ca/robots.txt
32 588 http://sigmdel.ca/michel/ha/rpi/streaming_en.html
33 465 http://sigmdel.ca/favicon.ico
34 455 http://sigmdel.ca/michel/index_en.html
35 438 http://sigmdel.ca/michel/ha/sonoff/img/sonoff-rpi.jpg
36 390 http://sigmdel.ca/michel/no-arrow-left.png
37 366 http://sigmdel.ca/
38 317 http://sigmdel.ca/michel/ha/rpi/img/mjpg_streamer_service_start.jpg
39 315 http://sigmdel.ca/michel/ha/rpi/img/mjpg_streamer_https.jpg
40 315 http://sigmdel.ca/michel/ha/rpi/img/mjpg_streamer_domoticz.jpg
41 315 http://sigmdel.ca/michel/ha/rpi/img/mjpg_streamer_https_chromium.jpg
42 314 http://sigmdel.ca/michel/ha/rpi/img/logitech-c270.jpg
43 313 http://sigmdel.ca/michel/ha/rpi/img/mjpg_streamer_login.jpg
44 292 http://sigmdel.ca/michel/ha/rpi/img/domoticz_webcam_01.jpg
45 291 http://sigmdel.ca/michel/ha/domo/img/install_domo_26.jpg
46 284 http://sigmdel.ca/michel/ha/rpi/img/domoticz_webcam_02.jpg
47 281 http://sigmdel.ca/michel/program/esp8266/arduino/watchdogs_en.html
48 277 http://sigmdel.ca/michel/ha/rpi/spi_on_pi_en.html
49 268 http://sigmdel.ca/michel/ha/xiao/seeeduino_xiao_01_en.html
50 268 http://sigmdel.ca/michel/ha/domo/img/install_domo_21.jpg
51 264 http://sigmdel.ca/michel/ha/domo/img/install_domo_25.jpg
52 264 http://sigmdel.ca/michel/ha/domo/img/install_domo_22.jpg
53 263 http://sigmdel.ca/michel/ha/domo/img/mqtt_publish_received.jpg
54 262 http://sigmdel.ca/michel/ha/domo/img/install_domo_24.jpg
55 261 http://sigmdel.ca/michel/ha/domo/img/mqtt_subscribe.jpg
56 261 http://sigmdel.ca/michel/ha/domo/img/install_domo_23.jpg
57 244 http://sigmdel.ca/michel/ha/rpi/img/rpi_spi0n1.png
58 240 http://sigmdel.ca/michel/ha/rpi/img/spi_rpi_esp8266_lcd.jpg
59 233 http://sigmdel.ca/michel/en-icon.png
60 233 http://sigmdel.ca/michel/program/esp8266/arduino/img/watchdog-01.png
61 232 http://sigmdel.ca/michel/ha/rpi/img/spi_mode_2.png
62 227 http://sigmdel.ca/michel/ha/rpi/img/spi_mode_0.png
63 226 http://sigmdel.ca/michel/program/esp8266/arduino/img/sw_watchdog_01.jpg
64 226 http://sigmdel.ca/michel/program/esp8266/arduino/img/hw_watchdog_03.jpg
65 225 http://sigmdel.ca/michel/program/esp8266/arduino/img/sw_watchdog_02.jpg
66 225 http://sigmdel.ca/michel/ha/rpi/img/spidev_bufsize4096.jpg
67 225 http://sigmdel.ca/michel/ha/rpi/img/spidev_bufsize1024.jpg
68 225 http://sigmdel.ca/michel/ha/rpi/img/test_logic_2.png
69 224 http://sigmdel.ca/michel/ha/rpi/img/spidev_timing.jpg
70 224 http://sigmdel.ca/michel/program/esp8266/arduino/img/hw_watchdog_02.jpg
71 224 http://sigmdel.ca/michel/ha/rpi/img/test_logic_1.png
72 223 http://sigmdel.ca/michel/program/esp8266/arduino/img/hw_watchdog_01.jpg
73 223 http://sigmdel.ca/michel/ha/rpi/img/spidev_bufsize8192.jpg
74 221 http://sigmdel.ca/michel/ha/rpi/img/spi_mode0.png
75 221 http://sigmdel.ca/michel/ha/rpi/img/spi_mode3.png
76 220 http://sigmdel.ca/michel/ha/rpi/img/spi_mode1.png
77 220 http://sigmdel.ca/michel/ha/rpi/img/spi_mode2.png
78 215 http://sigmdel.ca/michel/ha/wireguard/wireguard_omv_01_en.html
79 209 http://sigmdel.ca/michel/ha/domo/domo_03_en.html
80 203 http://sigmdel.ca/michel/ha/domo/img/mqtt-1_en.png
81 197 http://sigmdel.ca/michel/ha/domo/img/mqtt-2_en.png
82 197 http://sigmdel.ca/michel/ha/rpi/dnld/draft_spidev_doc.pdf
83 193 http://sigmdel.ca/michel/no-arrow-right.png
84 192 http://sigmdel.ca/michel/ha/rpi/img/alsamixer_01.jpg
85 189 http://sigmdel.ca/michel/ha/rpi/bluetooth_n_buster_01_en.html
86 170 http://sigmdel.ca/michel/ha/opi/img/OPiZ_uart.png
87 139 http://sigmdel.ca/michel/ha/domo/img/mqtt-1_fr.png
88 135 http://sigmdel.ca/michel/ha/opi/OPiZ_uart_en.html
89 134 http://sigmdel.ca/michel/ha/sonoff/img/sonoff_mini_opened.jpg
90 130 http://sigmdel.ca/michel/ha/rpi/bluetooth_02_en.html
91 130 http://sigmdel.ca/michel/ha/xiao/seeeduino_xiao_sercom_en.html
92 128 http://sigmdel.ca/michel/ha/esp8266/img/esp32radio_config_04.jpg
93 128 http://sigmdel.ca/michel/ha/sonoff/img/sonoff_basic_R3_diy_jumper.jpg
94 121 http://sigmdel.ca/michel/ha/xiao/img/seeeduino_xiao_01.jpg
95 121 http://sigmdel.ca/michel/ha/domo/img/mqtt-2_fr.png
96 118 http://sigmdel.ca/aed-dea/
97 116 http://sigmdel.ca/michel/ha/sonoff/sonoff_mini_en.html
98 114 http://sigmdel.ca/michel/ha/xiao/img/seeeduino_xiao_02.jpg
99 113 http://sigmdel.ca/michel/ha/xiao/img/xiao_uf2.jpg
100 112 http://sigmdel.ca/michel/ha/esp8266/img/esp32radio_config_01.jpg
************************************************************************
Results for module HTMLValidator
************************************************************************
Here are the 4 most popular invalid document(s) that I could find in the
logs for sigmdel.ca.
Rank Hits #Error(s) Address
------ ------ ----------- ------------------------------------------
121 18 15 http://sigmdel.ca/aed-dea/install2-en.html
152 12 4 http://sigmdel.ca/aed-dea/install-en.html
176 10 10 http://sigmdel.ca/aed-dea/install3-en.html
271 3 19 http://sigmdel.ca/meteo/ressources/
Conclusion :
You asked for 10 invalid HTML document but I could only find 4
by processing (all the) 295 document(s) in your logs.
This means that about 1.35% of your most popular documents were invalid.
************************************************************************
************************************************************************
Results for module CSSValidator
************************************************************************
Here is the most popular invalid document that I could find in the
logs for sigmdel.ca.
Rank Hits #Error(s) Address
------ ------ ----------- --------------------------------------
1 3089 1 http://sigmdel.ca/michel/css/base2.css
Conclusion :
You asked for 10 invalid stylesheet document(s) but I could only find 1
by processing (all the) 14 document(s) in your logs.
This means that about 7.14% of your most popular documents were invalid.
************************************************************************
Well that was a bit of a letdown. None of the most popular invalid documents are on this site at sigmdel.ca/michel/. Instead they are in two other sites that have to do with a former life when I was a, more or less, productive member of society. I should look at those four pages which I have gleefully ignored for years.