aa7c3b6852
Remove gravity optimization. Further investigations have shown that it is useless because the standard gravity run already produces an optimal database.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-09-07 20:50:11 +02:00
bb1a6243d6
Merge pull request #3669 from pi-hole/release/v5.1.2
...
Release/v5.1.2 to development
2020-08-14 16:28:53 +01:00
27399a762a
Check for compression onyl once and print result
...
Signed-off-by: Christian König <ckoenig@posteo.de>
2020-08-03 22:46:14 +02:00
87da9084e6
Use compression (if available) when downloading the ad lists.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-08-02 16:21:58 +01:00
527fc3c5cf
Also pidof -> pgrep in gravity.sh
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-07-21 22:03:54 +02:00
73963fecda
Use gravity's adlist_id in filename when saving downloaded adlist locally
...
Signed-off-by: Christian König <ckoenig@posteo.de>
2020-07-05 14:32:33 +02:00
dd205c55c1
Updated gravity.sh to allow semicolons
...
Signed-off-by: Peter Lowe <pgl@yoyo.org>
2020-05-26 15:53:10 +02:00
9e96ff37de
Fix merge conflicts
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2020-05-12 22:16:43 +01:00
288d487fc0
Allow ( and ) in adlist URLs.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-04-26 09:33:09 +02:00
fa57c457f3
Boldify number of unique domains as this is the actually interesting number
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-04-21 09:10:21 +02:00
0f20470a38
Merge pull request #3269 from pi-hole/tweak/hosts-comments
...
Add support for comments in HOSTS-like files
2020-04-15 11:11:10 -07:00
982c1b0059
Merge branch 'release/v5.0' into development
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-04-08 12:02:53 +02:00
7d19ee1b25
validate blocklist URL before adding to the database ( #3237 )
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
Co-authored-by: DL6ER <dl6er@dl6er.de>
2020-03-31 21:48:10 +01:00
7b15a88dc4
Strip comments from downloaded lists instead of discarding lines with comments altogether
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-03-31 18:36:40 +00:00
7b8611ced0
Assorted typo fixes. ( #3126 )
...
* Assorted typo fixes.
Signed-off-by: XhmikosR <xhmikosr@gmail.com>
Co-authored-by: Dan Schaper <dan.schaper@pi-hole.net>
2020-03-08 16:53:14 -07:00
3dd05606ca
Call it the received number of domains instead of the imported number as importing does only happen a bit later. Only show the number of invalid domains if there are invalid domains.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-24 07:06:15 +01:00
1e8bfd33f5
Improve output
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-23 22:50:06 +01:00
8131b5961c
Add comments to the code describing the changes.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-22 15:22:29 +01:00
81d4531e10
Implement performant list checking routine.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-22 13:01:55 +01:00
050e2963c7
Remove redundant code.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-21 22:28:53 +01:00
3c09cd4a3a
Experimental output of matching line from shown warnings.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-21 21:47:56 +01:00
839fe32042
Fix issue with missing newline at the end of adlists ( #3144 )
...
* Also display non-fatal warnings during the database importing. Previously, we have only show warnings when there were also errors (errors are always fatal).
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Ensure there is always a newline on the last line.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Stickler linting
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
* Move sed command into subroutine to avoid code duplication.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Also unify comments.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Also unify comments.
Signed-off-by: DL6ER <dl6er@dl6er.de>
Co-authored-by: Dan Schaper <dan@glacialmagma.com>
2020-02-21 18:56:48 +00:00
d14ee26d6a
Merge pull request #3139 from pi-hole/fix/count_before_calling_FTL
...
Fix wrong number of blocking domains shown on the dashboard
2020-02-17 21:32:45 +01:00
52398052e9
Compute number of domains (and store it in the database) BEFORE calling FTL to re-read said value.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-17 21:07:48 +01:00
601f9048cd
Merge pull request #3130 from pi-hole/fix/gravity_updated_timestamp
...
Store gravity update timestamp only after database swapping
2020-02-17 06:07:05 +01:00
dc2fce8e1d
Store gravity update timestamp only after database swapping.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-12 19:26:25 +01:00
2a5cf221fa
Store number of distinct gravity domains in database after counting.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-02-02 23:46:33 +01:00
10c2dad48a
Improve gravity performance ( #3100 )
...
* Gravity performance improvements.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Do not move downloaded lists into migration_backup directory.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Do not (strictly) sort domains. Random-leaf access is faster than always-last-leaf access (on average).
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Append instead of overwrite gravity_new collection list.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Rename table gravity_new to gravity_temp to clarify that this is only an intermediate table.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Add timers for each of the calls to compute intense parts. They are to be removed before this finally hits the release/v5.0 branch.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Fix legacy list files import. It currently doesn't work when the gravity database has already been updated to using the single domainlist table.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Simplify database_table_from_file(), remove all to this function for gravity lost downloads.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Update gravity.db.sql to version 10 to have newle created databases already reflect the most recent state.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Create second gravity database and swap them on success. This has a number of advantages such as instantaneous gravity updates (as seen from FTL) and always available gravity blocking. Furthermore, this saves disk space as the old database is removed on completion.
* Add timing output for the database swapping SQLite3 call.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Explicitly generate index as a separate process.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Remove time measurements.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-24 09:39:13 -08:00
8f22203d24
Wait 30 seconds for obtaining a database lock instead of immediately failing if the database is busy.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-14 20:02:00 +01:00
948f4a8827
Ensure permissions and ownership of gravity.db are correctly set on each run of pihole -g. This would have prevented https://github.com/pi-hole/AdminLTE/issues/1077
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-16 09:55:46 +00:00
2cec9eaf65
Merge pull request #3033 from pi-hole/fix/duplicates_in_adlists
...
Remove duplicates from adlists before importing
2019-12-12 21:37:26 +01:00
779fe670f7
Show full URL during gravity download instead of only domain and file
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:29:44 +00:00
570a7a5c11
Use sort -u instead of uniq as it is guaranteed to be safe when doing inline file operations.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:17:59 +00:00
bd1b004d94
Remove possible duplicates found in lower-quality adlists
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:13:51 +00:00
d29947ba32
optimise gravity list inserts
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-12-09 22:30:41 +00:00
1f03faddef
shell check recomends
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-12-09 21:35:54 +00:00
3231e5c3ba
Address stickler requests.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-09 16:52:03 +00:00
0c5185f8ba
Also display how many unique domains we have caught in the event horizon.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-04 21:02:46 +00:00
037d52104a
New command "pihole -g -r" recreates gravity.db based on files backed up in /etc/pihole/migration_update. This is useful to restore a working version of the database when the user destroyed the original database. Also, update gravity.db to version 5 because of a fix we needed to implement.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-11-26 10:58:39 +01:00
756c99653e
Merge branch 'development' into new/internal-blocking
2019-10-03 12:01:27 +02:00
61a40c1b43
merge devel into 4.3.2 And Resolve merge conflicts
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-09-22 01:16:44 +01:00
a8af2e1837
Store domains without sorting and unifying them first. This allows us to preserve the relationship of the individual domains to the lists they came from.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-09-04 23:14:29 +02:00
8bef5dc805
remove n from -ne
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-09-03 23:56:23 +01:00
ad41bcca5a
Remove support for adblock style lists to prevent false positives
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-09-03 23:43:11 +01:00
ca8982494b
Store timestamp when the gravity table was last updated successfully. This fixes https://github.com/pi-hole/AdminLTE/issues/989
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-09-01 14:42:07 +02:00
1820c2c598
Merge branch 'development' into new/whitelist-regex-support
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-08-22 14:19:51 +02:00
c53be459c6
quick fix for when dig also returns a CNAME
...
Signed-off-by: Andreas <ryrun@online.de>
2019-08-21 04:37:06 -07:00
4371c9ba03
Ensure proper permissions are set for gravity.db after creation.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-08-05 21:20:07 +02:00
63230cb72d
quick fix for when dig also returns a CNAME
...
Signed-off-by: ryrun <ryrun@online.de>
2019-08-04 21:21:08 +02:00
5ff9052200
Review comments
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-07-09 11:41:44 +02:00
DL6ER
Adam Warner
Christian König
Peter Lowe
Dan Schaper
XhmikosR
DL6ER
Andreas