Bug | Official mirror seems to be down/overburdened since 4 months ago

[Sommerwiesel]

What is the problem you are experiencing?

I cannot download any of the files from the official mirrors.

I've been getting errors on my pi.hole, where I added one of the hosts lists, that it could not download the list for the past 4 months.

Both my pihole, my offshore-servers, my private computer and my phone via carrier network simply either cannot establish a connection or fail to finish the download.

How can we reproduce the problem?

To reproduce the problem...

1. Add new list to pihole (i.e. https://hosts.ubuntu101.co.za/domains.list) OR Try to download the list manually
2. Connection cannot be established at all OR Download fails OR Download is at below 10 KB/s

Tried from the following networks:

• Private Internet connection in Germany
• Dedicated Server in Ukraine
• VPS in Netherlands
• VPS in USA
• carrier network (phone) in Germany

What did you expect to happen?

I expected that...
It works

Is there a workaround?

Nope

Additional context

Screenshot-20250508-101849.png

Screenshot-20250508-102700.png

Screenshot-20250508-102957.png

Screenshot-20250508-103241.png

[JonnyTech]

You have access to the lists blocked, either by pihole, dns, hosts file, security software or another means. Check your environment(s).

[Sommerwiesel]

You have access to the lists blocked, either by pihole, dns, hosts file, security software or another means. Check your environment(s).

No I don't have access blocked.

Proof: Download sometimes works but then with ~10 KB/s:
Screenshot-20250508-101849.png

The server / host hosts.ubuntu101.co.za seems to be down / overburdened for a long time.

There are also past (closed) issues with problems like mine, such as:

#711
#693

I've also tried downloading the file from 5 different endpoints.
And I don't use snake-oil security software at all.

TLDR: I'm 99.9% sure the problem is not on my side.

[JonnyTech]

Indeed, I double checked my setup and I now remember that I switched to Github for my downloads for the same reason - ahh, my failing memory!

This is the command that I use to download the main hosts file:

cp /dev/null hosts && wget https://raw.githubusercontent.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist/master/hosts/hosts{0..99} -O ->> hosts

For Pi-hole domains you should be able to use this url:

cp /dev/null domains.list && wget https://raw.githubusercontent.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist/master/domains/domains{0..99}.list -O ->> domains.list

Or the IPs list which is a single file:

cp /dev/null ips.list && wget https://raw.githubusercontent.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist/master/ips/ips0.list -O ips.list

The above are for Linux, not sure if wget works on Windows, but the command is:

cp /dev/null hosts && wget https://raw.githubusercontent.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist/master/hosts.windows/hosts{0..99}.windows -O ->> hosts

[zcwitkowski]

I've been getting complaints from DNSNet on a couple of my Android devices about not being able to download this list.

At this moment I'm trying to download this list on Linux using:

wget -c -t 5 -T 60 --no-check-certificate https://hosts.ubuntu101.co.za/hosts

and the current attempt so far:

--2025-05-10 13:50:01-- https://hosts.ubuntu101.co.za/hosts
Resolving hosts.ubuntu101.co.za (hosts.ubuntu101.co.za)... 88.198.70.39, 2a01:4f8:140:5021::39, 2a01:4f8:140:5021::5
Connecting to hosts.ubuntu101.co.za (hosts.ubuntu101.co.za)|88.198.70.39|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 37175142 (35M), 34622705 (33M) remaining [application/octet-stream]
Saving to: ‘hosts’

hosts 29%[+++===========> ] 10.31M 18.0KB/s eta 19m 58s

which, I suppose, is better than nothing. Crank up the retries and timeout and let it run in the background I guess.

[zcwitkowski]

Had to go out for a bit. Came back and tried to resume the download and got this:

$ wget -c -t 5 -T 60 --no-check-certificate https://hosts.ubuntu101.co.za/hosts
--2025-05-10 16:00:57-- https://hosts.ubuntu101.co.za/hosts
Resolving hosts.ubuntu101.co.za (hosts.ubuntu101.co.za)... 88.198.70.39, 2a01:4f8:140:5021::39, 2a01:4f8:140:5021::5
Connecting to hosts.ubuntu101.co.za (hosts.ubuntu101.co.za)|88.198.70.39|:443... connected.
GnuTLS: Error in the pull function.
Unable to establish SSL connection.
chrisw@antix1:/Downloads
$ wget -c -t 5 -T 60 --no-check-certificate https://hosts.ubuntu101.co.za/hosts
--2025-05-10 16:03:32-- https://hosts.ubuntu101.co.za/hosts
Resolving hosts.ubuntu101.co.za (hosts.ubuntu101.co.za)... 88.198.70.39, 2a01:4f8:140:5021::39, 2a01:4f8:140:5021::5
Connecting to hosts.ubuntu101.co.za (hosts.ubuntu101.co.za)|88.198.70.39|:443... connected.
GnuTLS: Error in the pull function.
Unable to establish SSL connection.
chrisw@antix1:/Downloads
$ wget -c -t 5 -T 60 --no-check-certificate https://hosts.ubuntu101.co.za/hosts
--2025-05-10 16:07:16-- https://hosts.ubuntu101.co.za/hosts
Resolving hosts.ubuntu101.co.za (hosts.ubuntu101.co.za)... 88.198.70.39, 2a01:4f8:140:5021::39, 2a01:4f8:140:5021::5
Connecting to hosts.ubuntu101.co.za (hosts.ubuntu101.co.za)|88.198.70.39|:443... connected.
Unable to establish SSL connection.
chrisw@antix1:~/Downloads
$ wget -c -t 5 -T 120 --no-check-certificate https://hosts.ubuntu101.co.za/hosts
--2025-05-10 16:09:24-- https://hosts.ubuntu101.co.za/hosts
Resolving hosts.ubuntu101.co.za (hosts.ubuntu101.co.za)... 88.198.70.39, 2a01:4f8:140:5021::39, 2a01:4f8:140:5021::5
Connecting to hosts.ubuntu101.co.za (hosts.ubuntu101.co.za)|88.198.70.39|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 37175162 (35M), 9360350 (8.9M) remaining [application/octet-stream]
Saving to: ‘hosts’

hosts 100%[+++++++++++++++++++++++++++++++++++++++=============>] 35.45M 22.1KB/s in 5m 14s

2025-05-10 16:14:43 (29.1 KB/s) - ‘hosts’ saved [37175162/37175162]

chrisw@antix1:~/Downloads
$

While it looks like it all got downloaded finally I believe the list is supposed to be updated between the time I started and finished, so, I will have to do it all over.

[Sommerwiesel]

@zcwitkowski
I don't think letting the file download with ridiculously low speed (for me its avg 10 KB/s) for up to 60 mins can be considered a solution.
Moreover, as you already experienced, the file might corrupt one way or another during said time.
For me, not a single download was able to finish.

I think what @JonnyTech wrote about manually concatenating the single files from github is a better solution.
I will set up my VPS for this.

Edit: @JonnyTech Your script does however check for every file from 0 - 99 and this causes a lot of unnecessary requests ending in 404. I will fix your snippets and upload a better version here.

[Sommerwiesel]

I will be temporarily serving all the files under my own mirror:

https://nerdvpn.de/blacklists/hosts
https://nerdvpn.de/blacklists/domains.list
https://nerdvpn.de/blacklists/hosts.windows
https://nerdvpn.de/blacklists/hosts.deny
https://nerdvpn.de/blacklists/superhosts.deny
https://nerdvpn.de/blacklists/ips.list
https://nerdvpn.de/blacklists/domains-dotted-format.list

And for #712:
https://nerdvpn.de/blacklists/hosts_compressed

Important info

• Download speed is limited to 2MB/s and only two simultaneous connections.
• I have disabled all logging for these URLS.
• The blacklists will be updated daily at 03:00 CEST. The update does not interfere with incoming download requests
• Excessive access and/or using weird/botlike user-agents will result in rate limiting and/or temporary lockout. Sorry but I have to protect everything internet thesedays from bots, bad actors and AI crawlers. Pihole and wget is fine.

Feel free to use them until they've fixed their official mirror.
Important: this is not an official mirror, no guarantee and I might take it down at any time.

For anyone interested in hosting their own mirror
The script to update these files is here:
https://paste.nerdvpn.de/?557cf315a3adea35#3BTUvW46sX71xryWB2syzKs6NJk71qxAwX5CeJP1pgyN

@mitchellkrogza @funilrys
If you want this to become an official alternative mirror, give me a PM on Matrix under @weidenwiesel:nerdvpn.de
Uptime stats here: https://updown.io/1yqh

[zcwitkowski]

JohnyTech's workaround doesn't work. I think it did work the first time I tried it, but, at present. I get only half (538981 entries) of what I should get.

I tried wget on your hosts file and got 403 Forbidden. I clicked on the link and got a 16.6MB file which is far short of the 35MB (1097810 entries) you get if the regular download completes.

Should be over a million entries according to https://github.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist.

The painfully slow command I currently use, and it may need to be issued multiple times, is:

wget -c -t 0 -T 120 --no-check-certificate https://hosts.ubuntu101.co.za/hosts

There's a few days old version at chrisw.freeshell.org/uhb/hosts, and, no, this should not be viewed as any kind of mirror. I will try to update it when and as I can, except, as I look at the progress of the current download it is now also showing 17MB instead of 35MB, so, no way to know when that can happen.

At this point maybe you grab the full copy off my web site to put on yours and then the question is how does all this get fixed? Who do you call?

[Sommerwiesel]

@zcwitkowski Ah yeah sorry my server completely rejects all machine-like user agents, including wget for obvious "fuck bots, abusive actors and AI scraper" reasons.
I forgot to remove that for those URLs, I'll do that now.
update: done, wget will work now

Thanks for the report on that :)

As for the length, they removed those "missing entries" along with some sources in a commit two days ago:
5cb98e5

The github readme now also lists "Total Bad domains in hosts file: 538,981" and not 1097810.

Don't know why, probably optimization or removal of obsolete entries, but 538981 lines is exactly how it should be.

Consequently, my mirror works perfectly and I still offer it for public usage.
Cheers