Commit Graph

422 Commits

Author SHA1 Message Date
DL6ER 6b33b8b4c0
Store status of downloaded list (downloaded, using cache, some error, ...) and number of (in-)valid domains on this list in the gravity database. This updates the gravity databaes to version 14.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-12-28 08:08:49 +01:00
user 922f795fe6 Adding comma to the adlist table format
To account for the Date Updated column in the adlist table adding a comma to the adlist to be imported into the database. This was causing intermittent failures on updates and therefore no domains would be blocked.

Signed-off-by: user <user@fedbrew.home>

 Date:      Sun Dec 6 06:03:30 2020 -0500
 Committer: user <user@fedbrew.home>

 On branch fix/gravity_date_updated
 Your branch and 'origin/fix/gravity_date_updated' have diverged,
 and have 1 and 1 different commits each, respectively.
   (use "git pull" to merge the remote branch into yours)

 Changes to be committed:
	modified:   gravity.sh

Signed-off-by: user <user@fedbrew.home>
2020-12-06 06:08:30 -05:00
Adam Warner 948337dd8e
Merge pull request #3740 from pi-hole/new/adlist_date_updated
Add date_updated field to adlist table
2020-10-09 14:04:13 +01:00
DL6ER 4fd412d7c3
Add date_updated field in adlist table set when a list changes.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-09-07 20:59:23 +02:00
DL6ER aa7c3b6852
Remove gravity optimization. Further investigations have shown that it is useless because the standard gravity run already produces an optimal database.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-09-07 20:50:11 +02:00
Adam Warner bb1a6243d6
Merge pull request #3669 from pi-hole/release/v5.1.2
Release/v5.1.2 to development
2020-08-14 16:28:53 +01:00
Christian König 27399a762a Check for compression onyl once and print result
Signed-off-by: Christian König <ckoenig@posteo.de>
2020-08-03 22:46:14 +02:00
DL6ER 87da9084e6
Use compression (if available) when downloading the ad lists.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-08-02 16:21:58 +01:00
DL6ER 527fc3c5cf
Also pidof -> pgrep in gravity.sh
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-07-21 22:03:54 +02:00
Christian König 73963fecda Use gravity's adlist_id in filename when saving downloaded adlist locally
Signed-off-by: Christian König <ckoenig@posteo.de>
2020-07-05 14:32:33 +02:00
Peter Lowe dd205c55c1 Updated gravity.sh to allow semicolons
Signed-off-by: Peter Lowe <pgl@yoyo.org>
2020-05-26 15:53:10 +02:00
Adam Warner 9e96ff37de
Fix merge conflicts
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2020-05-12 22:16:43 +01:00
DL6ER 288d487fc0
Allow ( and ) in adlist URLs.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-04-26 09:33:09 +02:00
DL6ER fa57c457f3
Boldify number of unique domains as this is the actually interesting number
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-04-21 09:10:21 +02:00
Dan Schaper 0f20470a38
Merge pull request #3269 from pi-hole/tweak/hosts-comments
Add support for comments in HOSTS-like files
2020-04-15 11:11:10 -07:00
DL6ER 982c1b0059
Merge branch 'release/v5.0' into development
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-04-08 12:02:53 +02:00
Adam Warner 7d19ee1b25
validate blocklist URL before adding to the database (#3237)
Signed-off-by: Adam Warner <me@adamwarner.co.uk>

Co-authored-by: DL6ER <dl6er@dl6er.de>
2020-03-31 21:48:10 +01:00
DL6ER 7b15a88dc4
Strip comments from downloaded lists instead of discarding lines with comments altogether
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-03-31 18:36:40 +00:00
XhmikosR 7b8611ced0
Assorted typo fixes. (#3126)
* Assorted typo fixes.

Signed-off-by: XhmikosR <xhmikosr@gmail.com>

Co-authored-by: Dan Schaper <dan.schaper@pi-hole.net>
2020-03-08 16:53:14 -07:00
DL6ER 3dd05606ca
Call it the received number of domains instead of the imported number as importing does only happen a bit later. Only show the number of invalid domains if there are invalid domains.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-24 07:06:15 +01:00
DL6ER 1e8bfd33f5
Improve output
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-23 22:50:06 +01:00
DL6ER 8131b5961c
Add comments to the code describing the changes.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-22 15:22:29 +01:00
DL6ER 81d4531e10
Implement performant list checking routine.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-22 13:01:55 +01:00
DL6ER 050e2963c7
Remove redundant code.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-21 22:28:53 +01:00
DL6ER 3c09cd4a3a
Experimental output of matching line from shown warnings.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-21 21:47:56 +01:00
DL6ER 839fe32042
Fix issue with missing newline at the end of adlists (#3144)
* Also display non-fatal warnings during the database importing. Previously, we have only show warnings when there were also errors (errors are always fatal).

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Ensure there is always a newline on the last line.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Stickler linting

Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>

* Move sed command into subroutine to avoid code duplication.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Also unify comments.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Also unify comments.

Signed-off-by: DL6ER <dl6er@dl6er.de>

Co-authored-by: Dan Schaper <dan@glacialmagma.com>
2020-02-21 18:56:48 +00:00
DL6ER d14ee26d6a
Merge pull request #3139 from pi-hole/fix/count_before_calling_FTL
Fix wrong number of blocking domains shown on the dashboard
2020-02-17 21:32:45 +01:00
DL6ER 52398052e9
Compute number of domains (and store it in the database) BEFORE calling FTL to re-read said value.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-17 21:07:48 +01:00
DL6ER 601f9048cd
Merge pull request #3130 from pi-hole/fix/gravity_updated_timestamp
Store gravity update timestamp only after database swapping
2020-02-17 06:07:05 +01:00
DL6ER dc2fce8e1d
Store gravity update timestamp only after database swapping.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-12 19:26:25 +01:00
DL6ER 2a5cf221fa
Store number of distinct gravity domains in database after counting.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-02 23:46:33 +01:00
DL6ER 10c2dad48a Improve gravity performance (#3100)
* Gravity performance improvements.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Do not move downloaded lists into migration_backup directory.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Do not (strictly) sort domains. Random-leaf access is faster than always-last-leaf access (on average).

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Append instead of overwrite gravity_new collection list.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Rename table gravity_new to gravity_temp to clarify that this is only an intermediate table.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Add timers for each of the calls to compute intense parts. They are to be removed before this finally hits the release/v5.0 branch.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Fix legacy list files import. It currently doesn't work when the gravity database has already been updated to using the single domainlist table.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Simplify database_table_from_file(), remove all to this function for gravity lost downloads.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Update gravity.db.sql to version 10 to have newle created databases already reflect the most recent state.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Create second gravity database and swap them on success. This has a number of advantages such as instantaneous gravity updates (as seen from FTL) and always available gravity blocking. Furthermore, this saves disk space as the old database is removed on completion.

* Add timing output for the database swapping SQLite3 call.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Explicitly generate index as a separate process.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Remove time measurements.

Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-24 09:39:13 -08:00
DL6ER 8f22203d24
Wait 30 seconds for obtaining a database lock instead of immediately failing if the database is busy.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-14 20:02:00 +01:00
DL6ER 948f4a8827
Ensure permissions and ownership of gravity.db are correctly set on each run of pihole -g. This would have prevented https://github.com/pi-hole/AdminLTE/issues/1077
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-16 09:55:46 +00:00
DL6ER 2cec9eaf65
Merge pull request #3033 from pi-hole/fix/duplicates_in_adlists
Remove duplicates from adlists before importing
2019-12-12 21:37:26 +01:00
DL6ER 779fe670f7
Show full URL during gravity download instead of only domain and file
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:29:44 +00:00
DL6ER 570a7a5c11
Use sort -u instead of uniq as it is guaranteed to be safe when doing inline file operations.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:17:59 +00:00
DL6ER bd1b004d94
Remove possible duplicates found in lower-quality adlists
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:13:51 +00:00
Adam Warner d29947ba32
optimise gravity list inserts
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-12-09 22:30:41 +00:00
Adam Warner 1f03faddef
shell check recomends
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-12-09 21:35:54 +00:00
DL6ER 3231e5c3ba
Address stickler requests.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-09 16:52:03 +00:00
DL6ER 0c5185f8ba
Also display how many unique domains we have caught in the event horizon.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-04 21:02:46 +00:00
DL6ER 037d52104a
New command "pihole -g -r" recreates gravity.db based on files backed up in /etc/pihole/migration_update. This is useful to restore a working version of the database when the user destroyed the original database. Also, update gravity.db to version 5 because of a fix we needed to implement.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-11-26 10:58:39 +01:00
DL6ER 756c99653e
Merge branch 'development' into new/internal-blocking 2019-10-03 12:01:27 +02:00
Adam Warner 61a40c1b43
merge devel into 4.3.2 And Resolve merge conflicts
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-09-22 01:16:44 +01:00
DL6ER a8af2e1837
Store domains without sorting and unifying them first. This allows us to preserve the relationship of the individual domains to the lists they came from.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-09-04 23:14:29 +02:00
Adam Warner 8bef5dc805
remove n from -ne
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-09-03 23:56:23 +01:00
Adam Warner ad41bcca5a
Remove support for adblock style lists to prevent false positives
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-09-03 23:43:11 +01:00
DL6ER ca8982494b
Store timestamp when the gravity table was last updated successfully. This fixes https://github.com/pi-hole/AdminLTE/issues/989
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-09-01 14:42:07 +02:00
DL6ER 1820c2c598
Merge branch 'development' into new/whitelist-regex-support
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-08-22 14:19:51 +02:00