Commit Graph

425 Commits

Author SHA1 Message Date
DL6ER 0944807491
Actually store correct number of domains for the individual lists (and not the sum of the so far collected number of domains)
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-12-29 20:36:49 +01:00
DL6ER a216848c1d
Only update time of last list change when we see a list for the first time or when it really changed content
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-12-29 20:28:09 +01:00
DL6ER 8c56f54a1e
Compare checksum of downloaded list against older checksums to see if the list content changed since the last download
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-12-29 09:54:25 +01:00
DL6ER 6b33b8b4c0
Store status of downloaded list (downloaded, using cache, some error, ...) and number of (in-)valid domains on this list in the gravity database. This updates the gravity databaes to version 14.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-12-28 08:08:49 +01:00
user 922f795fe6 Adding comma to the adlist table format
To account for the Date Updated column in the adlist table adding a comma to the adlist to be imported into the database. This was causing intermittent failures on updates and therefore no domains would be blocked.

Signed-off-by: user <user@fedbrew.home>

 Date:      Sun Dec 6 06:03:30 2020 -0500
 Committer: user <user@fedbrew.home>

 On branch fix/gravity_date_updated
 Your branch and 'origin/fix/gravity_date_updated' have diverged,
 and have 1 and 1 different commits each, respectively.
   (use "git pull" to merge the remote branch into yours)

 Changes to be committed:
	modified:   gravity.sh

Signed-off-by: user <user@fedbrew.home>
2020-12-06 06:08:30 -05:00
Adam Warner 948337dd8e
Merge pull request #3740 from pi-hole/new/adlist_date_updated
Add date_updated field to adlist table
2020-10-09 14:04:13 +01:00
DL6ER 4fd412d7c3
Add date_updated field in adlist table set when a list changes.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-09-07 20:59:23 +02:00
DL6ER aa7c3b6852
Remove gravity optimization. Further investigations have shown that it is useless because the standard gravity run already produces an optimal database.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-09-07 20:50:11 +02:00
Adam Warner bb1a6243d6
Merge pull request #3669 from pi-hole/release/v5.1.2
Release/v5.1.2 to development
2020-08-14 16:28:53 +01:00
Christian König 27399a762a Check for compression onyl once and print result
Signed-off-by: Christian König <ckoenig@posteo.de>
2020-08-03 22:46:14 +02:00
DL6ER 87da9084e6
Use compression (if available) when downloading the ad lists.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-08-02 16:21:58 +01:00
DL6ER 527fc3c5cf
Also pidof -> pgrep in gravity.sh
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-07-21 22:03:54 +02:00
Christian König 73963fecda Use gravity's adlist_id in filename when saving downloaded adlist locally
Signed-off-by: Christian König <ckoenig@posteo.de>
2020-07-05 14:32:33 +02:00
Peter Lowe dd205c55c1 Updated gravity.sh to allow semicolons
Signed-off-by: Peter Lowe <pgl@yoyo.org>
2020-05-26 15:53:10 +02:00
Adam Warner 9e96ff37de
Fix merge conflicts
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2020-05-12 22:16:43 +01:00
DL6ER 288d487fc0
Allow ( and ) in adlist URLs.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-04-26 09:33:09 +02:00
DL6ER fa57c457f3
Boldify number of unique domains as this is the actually interesting number
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-04-21 09:10:21 +02:00
Dan Schaper 0f20470a38
Merge pull request #3269 from pi-hole/tweak/hosts-comments
Add support for comments in HOSTS-like files
2020-04-15 11:11:10 -07:00
DL6ER 982c1b0059
Merge branch 'release/v5.0' into development
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-04-08 12:02:53 +02:00
Adam Warner 7d19ee1b25
validate blocklist URL before adding to the database (#3237)
Signed-off-by: Adam Warner <me@adamwarner.co.uk>

Co-authored-by: DL6ER <dl6er@dl6er.de>
2020-03-31 21:48:10 +01:00
DL6ER 7b15a88dc4
Strip comments from downloaded lists instead of discarding lines with comments altogether
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-03-31 18:36:40 +00:00
XhmikosR 7b8611ced0
Assorted typo fixes. (#3126)
* Assorted typo fixes.

Signed-off-by: XhmikosR <xhmikosr@gmail.com>

Co-authored-by: Dan Schaper <dan.schaper@pi-hole.net>
2020-03-08 16:53:14 -07:00
DL6ER 3dd05606ca
Call it the received number of domains instead of the imported number as importing does only happen a bit later. Only show the number of invalid domains if there are invalid domains.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-24 07:06:15 +01:00
DL6ER 1e8bfd33f5
Improve output
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-23 22:50:06 +01:00
DL6ER 8131b5961c
Add comments to the code describing the changes.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-22 15:22:29 +01:00
DL6ER 81d4531e10
Implement performant list checking routine.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-22 13:01:55 +01:00
DL6ER 050e2963c7
Remove redundant code.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-21 22:28:53 +01:00
DL6ER 3c09cd4a3a
Experimental output of matching line from shown warnings.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-21 21:47:56 +01:00
DL6ER 839fe32042
Fix issue with missing newline at the end of adlists (#3144)
* Also display non-fatal warnings during the database importing. Previously, we have only show warnings when there were also errors (errors are always fatal).

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Ensure there is always a newline on the last line.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Stickler linting

Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>

* Move sed command into subroutine to avoid code duplication.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Also unify comments.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Also unify comments.

Signed-off-by: DL6ER <dl6er@dl6er.de>

Co-authored-by: Dan Schaper <dan@glacialmagma.com>
2020-02-21 18:56:48 +00:00
DL6ER d14ee26d6a
Merge pull request #3139 from pi-hole/fix/count_before_calling_FTL
Fix wrong number of blocking domains shown on the dashboard
2020-02-17 21:32:45 +01:00
DL6ER 52398052e9
Compute number of domains (and store it in the database) BEFORE calling FTL to re-read said value.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-17 21:07:48 +01:00
DL6ER 601f9048cd
Merge pull request #3130 from pi-hole/fix/gravity_updated_timestamp
Store gravity update timestamp only after database swapping
2020-02-17 06:07:05 +01:00
DL6ER dc2fce8e1d
Store gravity update timestamp only after database swapping.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-12 19:26:25 +01:00
DL6ER 2a5cf221fa
Store number of distinct gravity domains in database after counting.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-02 23:46:33 +01:00
DL6ER 10c2dad48a Improve gravity performance (#3100)
* Gravity performance improvements.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Do not move downloaded lists into migration_backup directory.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Do not (strictly) sort domains. Random-leaf access is faster than always-last-leaf access (on average).

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Append instead of overwrite gravity_new collection list.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Rename table gravity_new to gravity_temp to clarify that this is only an intermediate table.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Add timers for each of the calls to compute intense parts. They are to be removed before this finally hits the release/v5.0 branch.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Fix legacy list files import. It currently doesn't work when the gravity database has already been updated to using the single domainlist table.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Simplify database_table_from_file(), remove all to this function for gravity lost downloads.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Update gravity.db.sql to version 10 to have newle created databases already reflect the most recent state.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Create second gravity database and swap them on success. This has a number of advantages such as instantaneous gravity updates (as seen from FTL) and always available gravity blocking. Furthermore, this saves disk space as the old database is removed on completion.

* Add timing output for the database swapping SQLite3 call.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Explicitly generate index as a separate process.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Remove time measurements.

Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-24 09:39:13 -08:00
DL6ER 8f22203d24
Wait 30 seconds for obtaining a database lock instead of immediately failing if the database is busy.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-14 20:02:00 +01:00
DL6ER 948f4a8827
Ensure permissions and ownership of gravity.db are correctly set on each run of pihole -g. This would have prevented https://github.com/pi-hole/AdminLTE/issues/1077
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-16 09:55:46 +00:00
DL6ER 2cec9eaf65
Merge pull request #3033 from pi-hole/fix/duplicates_in_adlists
Remove duplicates from adlists before importing
2019-12-12 21:37:26 +01:00
DL6ER 779fe670f7
Show full URL during gravity download instead of only domain and file
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:29:44 +00:00
DL6ER 570a7a5c11
Use sort -u instead of uniq as it is guaranteed to be safe when doing inline file operations.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:17:59 +00:00
DL6ER bd1b004d94
Remove possible duplicates found in lower-quality adlists
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:13:51 +00:00
Adam Warner d29947ba32
optimise gravity list inserts
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-12-09 22:30:41 +00:00
Adam Warner 1f03faddef
shell check recomends
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-12-09 21:35:54 +00:00
DL6ER 3231e5c3ba
Address stickler requests.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-09 16:52:03 +00:00
DL6ER 0c5185f8ba
Also display how many unique domains we have caught in the event horizon.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-04 21:02:46 +00:00
DL6ER 037d52104a
New command "pihole -g -r" recreates gravity.db based on files backed up in /etc/pihole/migration_update. This is useful to restore a working version of the database when the user destroyed the original database. Also, update gravity.db to version 5 because of a fix we needed to implement.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-11-26 10:58:39 +01:00
DL6ER 756c99653e
Merge branch 'development' into new/internal-blocking 2019-10-03 12:01:27 +02:00
Adam Warner 61a40c1b43
merge devel into 4.3.2 And Resolve merge conflicts
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-09-22 01:16:44 +01:00
DL6ER a8af2e1837
Store domains without sorting and unifying them first. This allows us to preserve the relationship of the individual domains to the lists they came from.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-09-04 23:14:29 +02:00
Adam Warner 8bef5dc805
remove n from -ne
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-09-03 23:56:23 +01:00