merge lan changes from master

README.md

@@ -11,9 +11,7 @@
     Anonymous social platform, mail, file sharing.
 </p>
 
-<img src='https://img.shields.io/github/license/beardog108/onionr'> <img src='https://gitlab.com/beardog/Onionr/badges/master/build.svg'> <img src='https://img.shields.io/badge/docker%20%F0%9F%90%8B-supported-success'> <img src='https://img.shields.io/badge/python%20version%20%F0%9F%90%8D-3.7+-blue'> <img src='https://img.shields.io/github/commit-activity/m/beardog108/onionr'>
+<img src='https://img.shields.io/github/license/beardog108/onionr'> <img src='https://gitlab.com/beardog/Onionr/badges/master/build.svg'> <img src='https://img.shields.io/badge/python%20version%20%F0%9F%90%8D-3.7+-blue'> <img src='https://img.shields.io/github/commit-activity/m/beardog108/onionr'>
-
-<img src='https://onionr.net/block-count.svg' alt='current stored block count'>
 
 <a href='https://www.reddit.com/r/onionr'><img src = 'https://img.shields.io/reddit/subreddit-subscribers/onionr?style=social'></a> <a href='https://twitter.com/onionrnet'><img src='https://img.shields.io/twitter/follow/onionrnet?style=social'></a>
 
@@ -25,16 +23,16 @@
 
 <hr>
 
-**The main repository for this software is at https://GitLab.com/beardog/Onionr/**
+**The main repository for this software is at https://git.VoidNet.tech/kev/onionr/**
 
-**The [GitHub repository](https://github.com/beardog108/onionr/) is a mirror, do not submit PRs or issues there.**
+Mirrors: [Github](https://github.com/beardog108/onionr), [Gitlab](https://gitlab.com/beardog/onionr)
 
 
 Onionr ("Onion Relay") is a decentralized/distributed peer-to-peer communication network, designed to be anonymous and resistant to (meta)data analysis, spam, and corruption.
 
-Onionr stores data in independent packages referred to as 'blocks'. The blocks are distributed to all interested nodes. Blocks and user IDs cannot be easily proven to have been created by a particular user. Even if there is enough evidence to believe that a specific user created a block, nodes still operate behind Tor or I2P and as such cannot be trivially unmasked. Anonymity is achieved by a stateless network, with no given indication of what node a block originates from. Through message mixing and key privacy, it is intended to be nigh impossible to discover the identity of a message creator or recipient.
+Onionr stores data in independent packages referred to as 'blocks'. The blocks are distributed to all interested nodes. Blocks and user IDs cannot be easily proven to have been created by a particular user. Even if there is enough evidence to believe that a specific user created a block, nodes still operate behind Tor and as such cannot be trivially unmasked. Anonymity is achieved by a stateless network, with no given indication of what node a block originates from. Through message mixing and key privacy, it is intended to be nigh impossible to discover the identity of a message creator or recipient.
 
-Via long-term traffic analysis, a well funded adversary may discover the most probable node(s) to be creating a set of related blocks, however doing so would only lead them to a node behind Tor or I2P. As the first node that a block appears on is almost always not the creator of the block, there is plausible deniability regarding the true creator of the block.
+Via long-term traffic analysis, a well funded adversary may discover the most probable node(s) to be creating a set of related blocks, however doing so would only lead them to a node behind Tor. As the first node that a block appears on is almost always not the creator of the block, there is plausible deniability regarding the true creator of the block.
 
 Onionr gives the individual the ability to speak freely, without fear of surveillance and censorship.
 
@@ -139,7 +137,7 @@ Everyone is welcome to contribute. Help is wanted for the following:
 * UI/UX design
 * Running stable nodes
 * Security review/audit
-* Automatic I2P setup
+* I2P support
 
 ## Contribute money:
 

docs/TODO.txt

@@ -12,7 +12,6 @@
 * make node "speed" setting such as when ui is open to reduce bandwidth usage
 * localization support
 
-* add mark read in mail
 * add BCC support to mail
 
 

docs/whitepaper.md

@@ -4,13 +4,15 @@
 
 <p align="center">Anonymous, Decentralized, Distributed Network</p>
 
+June 2020
+
 # Introduction
 
 We believe that the ability to communicate freely with others is crucial for maintaining societal and personal liberty. The internet has provided humanity with the ability to spread information globally, but there are many persons and organizations who try to stifle the flow of information, sometimes with success.
 
-Internet censorship comes in many forms, state censorship, corporate consolidation of media, threats of violence, network exploitation (e.g. denial of service attacks) and other threats.
+Internet censorship comes in many forms, state censorship, threats of violence, network exploitation (e.g. denial of service attacks) and others.
 
-We hold that in order to protect individual privacy, users must have the ability to communicate anonymously and with decentralization. 
+We hold that in order to protect individual privacy, users must have the ability to communicate anonymously and with decentralization.
 
 We believe that in order to prevent censorship and loss of information, these measures must be in place:
 
@@ -58,12 +60,14 @@ To mitigate maliciously slow or unreliable nodes, Onionr builds a profile on nod
 
 ## Block Format
 
-Onionr blocks are very simple. They are structured in two main parts: a metadata section and a data section, with a line feed delimiting where metadata ends and data begins. 
+Onionr blocks are very simple. They are structured in two main parts: a metadata section and a data section, with a line feed delimiting where metadata ends and data begins.
 
 Metadata defines what kind of data is in a block, signature data, encryption settings, and other arbitrary information.
 
 Optionally, a random token can be inserted into the metadata for use in Proof of Work.
 
+The proof of work function should be a Verifiable Delay Function (VDF). We have chosen MiMC for it's simplicity.
+
 ### Block Encryption
 
 For encryption, Onionr uses ephemeral Curve25519 keys for key exchange and XSalsa20-Poly1305 as a symmetric cipher or optionally using only XSalsa20-Poly1305 with a pre-shared key.
@@ -84,9 +88,7 @@ Blocks are stored indefinitely until the allocated space is filled, at which poi
 
 ## Block Timestamping
 
-Onionr can provide evidence of when a block was inserted by requesting other users to sign a hash of the current time with the block data hash: sha3_256(time + sha3_256(block data)).
+Onionr blocks are by default not accepted if their timestamp is set too far in the past, or is in the future.
-
-This can be done either by the creator of the block prior to generation, or by any node after insertion.
 
 In addition, randomness beacons such as the one operated by [NIST](https://beacon.nist.gov/home), [Chile](https://beacon.clcert.cl/), or the hash of the latest blocks in a cryptocurrency network could be used to affirm that a block was at least not *created* before a given time.
 
@@ -98,7 +100,7 @@ The benefits of such a system are increased privacy, and the ability to anonymou
 
 # Threat Model
 
-The goal of Onionr is to provide a method of distributing information in a manner in which the difficulty of discovering the identity of those sending and receiving the information is greatly increased. In this section we detail what information we want to protect and who we're protecting it from. 
+The goal of Onionr is to provide a method of distributing information in a manner in which the difficulty of discovering the identity of those sending and receiving the information is greatly increased. In this section we detail what information we want to protect and who we're protecting it from.
 
 In this threat model, "protected" means available in plaintext only to those which it was intended, and regardless non-malleable
 
@@ -132,7 +134,7 @@ Onionr does not protect the following:
 
 ## Assumptions
 
-We assume that Tor onion services (v3) and I2P services cannot be trivially deanonymized, and that the underlying cryptographic primitives we employ cannot be broken in any manner faster than brute force unless a quantum computer is used. 
+We assume that Tor onion services (v3) and I2P services cannot be trivially deanonymized, and that the underlying cryptographic primitives we employ cannot be broken in any manner faster than brute force unless a quantum computer is used.
 
 Once quantum safe algorithms are more mature and have decent high level libraries, they will be deployed.
 

requirements-dev.in

@@ -1,3 +1,3 @@
-pdoc3==0.7.5
+pdoc3==0.8.3
-pip-tools==4.5.1
+pip-tools==5.2.1
-helium==3.0.1
+helium==3.0.4

requirements-dev.txt

@@ -8,8 +8,9 @@ click==7.0 \
     --hash=sha256:2335065e6395b9e67ca716de5f7526736bfa6ceead690adf616d925bdc622b13 \
     --hash=sha256:5b94b49521f6456670fdb30cd82a4eca9412788a93fa6dd6df72c94d5a8ff2d7 \
     # via pip-tools
-helium==3.0.1 \
+helium==3.0.4 \
-    --hash=sha256:a7be1ad48702a38e11e9ee9b262459cde3c986d2dcbd79fcf8d6f5b54c5f5d9e
+    --hash=sha256:035edb4207906fde42f64d47e28dca934327f00fc23c808b45090c2266123998 \
+    # via -r requirements-dev.in
 mako==1.1.1 \
     --hash=sha256:2984a6733e1d472796ceef37ad48c26f4a984bb18119bb2dbc37a44d8f6e75a4 \
     # via pdoc3
@@ -52,26 +53,27 @@ markupsafe==1.1.1 \
     --hash=sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7 \
     --hash=sha256:e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be \
     # via mako
+pdoc3==0.8.3 \
+    --hash=sha256:19bd1a72e1c82875a6927b244aca826dedb635ea2a7a36ac62cbe063a8ddc30d \
+    # via -r requirements-dev.in
+pip-tools==5.2.1 \
+    --hash=sha256:1690bef5f0f714160c3aedacb03520e2359a78f7f9fa17e574cf8659cf2ef614 \
+    --hash=sha256:5b4b6e7b6e66357685c73e856296b4792b2d159ff6074729e250e291834bfd9d \
+    # via -r requirements-dev.in
 selenium==3.141.0 \
     --hash=sha256:2d7131d7bc5a5b99a2d9b04aaf2612c411b03b8ca1b1ee8d3de5845a9be2cb3c \
     --hash=sha256:deaf32b60ad91a4611b98d8002757f29e6f2c2d5fcaf202e1c9ad06d6772300d \
     # via helium
-pdoc3==0.7.5 \
-    --hash=sha256:ebca75b7fcf23f3b4320abe23339834d3f08c28517718e9d29e555fc38eeb33c \
-    # via -r requirements-dev.in
-pip-tools==4.5.1 \
-    --hash=sha256:693f30e451875796b1b25203247f0b4cf48a4c4a5ab7341f4f33ffd498cdcc98 \
-    --hash=sha256:be9c796aa88b2eec5cabf1323ba1cb60a08212b84bfb75b8b4037a8ef8cb8cb6 \
-    # via -r requirements-dev.in
 six==1.14.0 \
     --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \
     --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c \
     # via pip-tools
-urllib3==1.25.8 \
+urllib3==1.25.9 \
-    --hash=sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc \
+    --hash=sha256:3018294ebefce6572a474f0604c2021e33b3fd8006ecd11d62107a5d2a963527 \
-    --hash=sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc \
+    --hash=sha256:88206b0eb87e6d677d424843ac5209e3fb9d0190d0ee169599165ec25e9d9115 \
     # via selenium
 
 # WARNING: The following packages were not pinned, but pip requires them to be
 # pinned when the requirements file includes hashes. Consider using the --allow-unsafe flag.
+# pip
 # setuptools

requirements.in

@@ -1,16 +1,17 @@
-urllib3==1.25.8
+urllib3==1.25.9
-requests==2.23.0
+requests==2.24.0
-PyNaCl==1.3.0
+PyNaCl==1.4.0
-gevent==1.4.0
+gevent==20.6.2
-Flask==1.1.1
+Flask==1.1.2
 PySocks==1.7.1
 stem==1.8.0
 deadsimplekv==0.3.1
 unpaddedbase32==0.2.0
-streamedrequests==1.0.0
+streamedrequests==1.0.3
 toomanyobjs==1.1.0
 niceware==0.2.1
-psutil==5.7.0
+psutil==5.7.2
 filenuke==0.0.0
-mimcvdf==1.0.0
+mimcvdf==1.1.0
-watchdog==0.10.2
+watchdog==0.10.3
+ujson==3.0.0

requirements.txt

@@ -54,55 +54,58 @@ filenuke==0.0.0 \
     --hash=sha256:147011c0125121469cae0a8a7f4df399f470e54aa29a08f2d2c099bf0118dcee \
     --hash=sha256:c55535dcecfdb27c5f4ce664d46e115950b5429763b5db75c198053646177f8f \
     # via -r requirements.in
-flask==1.1.1 \
+flask==1.1.2 \
-    --hash=sha256:13f9f196f330c7c2c5d7a5cf91af894110ca0215ac051b5844701f2bfd934d52 \
+    --hash=sha256:4efa1ae2d7c9865af48986de8aeb8504bf32c7f3d6fdc9353d34b21f4b127060 \
-    --hash=sha256:45eb5a6fd193d6cf7e0cf5d8a5b31f83d5faae0293695626f539a823e93b13f6 \
+    --hash=sha256:8a4fdd8936eba2512e9c85df320a37e694c93945b33ef33c89946a340a238557 \
     # via -r requirements.in
-gevent==1.4.0 \
+gevent==20.6.2 \
-    --hash=sha256:0774babec518a24d9a7231d4e689931f31b332c4517a771e532002614e270a64 \
+    --hash=sha256:0b16dd85eddaf6acdad373ce90ed4da09ef466cbc5e0ee5932d13f099929e844 \
-    --hash=sha256:0e1e5b73a445fe82d40907322e1e0eec6a6745ca3cea19291c6f9f50117bb7ea \
+    --hash=sha256:0f3fbb1703b10609856e5dffb0e358bf5edf57e52dc7cd7226e3f8674fdc0a0f \
-    --hash=sha256:0ff2b70e8e338cf13bedf146b8c29d475e2a544b5d1fe14045aee827c073842c \
+    --hash=sha256:13c74d6784ef5ada2666abf2bb310d27a1d14291f7cac46148f336b19f714d40 \
-    --hash=sha256:107f4232db2172f7e8429ed7779c10f2ed16616d75ffbe77e0e0c3fcdeb51a51 \
+    --hash=sha256:1ea0d34cb78cdf37870be3bfb9330ebda89197bed9e048c14f4a90dec19a33e0 \
-    --hash=sha256:14b4d06d19d39a440e72253f77067d27209c67e7611e352f79fe69e0f618f76e \
+    --hash=sha256:354f932c284fa45826b32f42927d892096cce05671b50b3ff59528230217ad47 \
-    --hash=sha256:1b7d3a285978b27b469c0ff5fb5a72bcd69f4306dbbf22d7997d83209a8ba917 \
+    --hash=sha256:3cb2f6978615d52e4e4e667b035c11a7272bb68b14d119faf1b138164b2f354f \
-    --hash=sha256:1eb7fa3b9bd9174dfe9c3b59b7a09b768ecd496debfc4976a9530a3e15c990d1 \
+    --hash=sha256:67776cb33b638a3c61a0351d9d1e8f33a46b47de619e249de1159892f9ff035c \
-    --hash=sha256:2711e69788ddb34c059a30186e05c55a6b611cb9e34ac343e69cf3264d42fe1c \
+    --hash=sha256:68764aca061bbbbade43727e797f9c28042f6d90cca5fb6514ef726d43ab00ca \
-    --hash=sha256:28a0c5417b464562ab9842dd1fb0cc1524e60494641d973206ec24d6ec5f6909 \
+    --hash=sha256:6c864b5604166ac8351e3128a1135b883b9e978fd24afbd75a249dcb42bc8ab5 \
-    --hash=sha256:3249011d13d0c63bea72d91cec23a9cf18c25f91d1f115121e5c9113d753fa12 \
+    --hash=sha256:73eb4cf3114fbb5dd801bd0b93941adfa2fa6d99e91976c20a121ea14b8b39b9 \
-    --hash=sha256:44089ed06a962a3a70e96353c981d628b2d4a2f2a75ea5d90f916a62d22af2e8 \
+    --hash=sha256:76ef4c6e3332e6f7278142d791b28695adfce39735900fccef2a0f1d894f6b36 \
-    --hash=sha256:4bfa291e3c931ff3c99a349d8857605dca029de61d74c6bb82bd46373959c942 \
+    --hash=sha256:78bd94f6f2ac366155169df3507068f6381f2ad77625633189ce183f86a57597 \
-    --hash=sha256:50024a1ee2cf04645535c5ebaeaa0a60c5ef32e262da981f4be0546b26791950 \
+    --hash=sha256:7d8408854ce892f987305a0e9bf5c051f4ea29453665454396d6afb620c719b6 \
-    --hash=sha256:53b72385857e04e7faca13c613c07cab411480822ac658d97fd8a4ddbaf715c8 \
+    --hash=sha256:9527087984f1659be899b3300d5d61c7c5b01d8beae106aff5160316da8bc56f \
-    --hash=sha256:74b7528f901f39c39cdbb50cdf08f1a2351725d9aebaef212a29abfbb06895ee \
+    --hash=sha256:a18d8dd9bfa994a22f30adfa0563d80f0809140045c34f85535f422813d25855 \
-    --hash=sha256:7d0809e2991c9784eceeadef01c27ee6a33ca09ebba6154317a257353e3af922 \
+    --hash=sha256:a23c2abf08e851c988723f6a2996d495f513a2c0dc70f9956af03af8debdb5d1 \
-    --hash=sha256:896b2b80931d6b13b5d9feba3d4eebc67d5e6ec54f0cf3339d08487d55d93b0e \
+    --hash=sha256:a47556cac07e31b3cef8fd701599b3b1365961fe3736471f41807ffa27c5c848 \
-    --hash=sha256:8d9ec51cc06580f8c21b41fd3f2b3465197ba5b23c00eb7d422b7ae0380510b0 \
+    --hash=sha256:b03890bbddbae5667f5baad517417056496ff5e92c3c7945b27cc08f55a9fcb2 \
-    --hash=sha256:9f7a1e96fec45f70ad364e46de32ccacab4d80de238bd3c2edd036867ccd48ad \
+    --hash=sha256:b17915b65b49a425115ddc3087484c81b1e47ce38c931d18bb14e453753e4d06 \
-    --hash=sha256:ab4dc33ef0e26dc627559786a4fba0c2227f125db85d970abbf85b77506b3f51 \
+    --hash=sha256:bef18b8bd3b728240b9bbd699737216b793d6c97b482431f69dcbe328ad73692 \
-    --hash=sha256:d1e6d1f156e999edab069d79d890859806b555ce4e4da5b6418616322f0a3df1 \
+    --hash=sha256:c0f4340e40e0f9dfe93a52a12ddf5b1eeda9bbc89b99bf3b9b23acab0dfae0a4 \
-    --hash=sha256:d752bcf1b98174780e2317ada12013d612f05116456133a6acf3e17d43b71f05 \
+    --hash=sha256:d0a67a20ce325f6a2068e0bd9fbf83db8a5f5ced972ed8ac5c20079a7d98c7d1 \
-    --hash=sha256:e5bcc4270671936349249d26140c267397b7b4b1381f5ec8b13c53c5b53ab6e1 \
+    --hash=sha256:d3baff87d935a5eeffb0e4f7cd5ffe258d2430cd62aeee2e5396f85da07df435 \
+    --hash=sha256:e5ca5ee80a9d9e697c9fc22b4bbce9ad06870f83fc8e7774e5504892ef702476 \
+    --hash=sha256:ea2e4584950186b71d648bde6af40dae4d4c6f43db25a732ec056b27a7a83afe \
+    --hash=sha256:ebb8a545112110e3a6edf905ae1556b0538fc148c743aa7d8cfaebbbc23de31d \
+    --hash=sha256:f2a02d9004ccb18edd9eaf6f25da9a7763de41a69754d5e4d872a8cbf8bd0b72 \
+    --hash=sha256:f41cc8e853ac2252bc58f6feabd74b8aae613e2d19097c5373463122f4dc08f0 \
     # via -r requirements.in
-greenlet==0.4.15 \
+greenlet==0.4.16 \
-    --hash=sha256:000546ad01e6389e98626c1367be58efa613fa82a1be98b0c6fc24b563acc6d0 \
+    --hash=sha256:1000038ba0ea9032948e2156a9c15f5686f36945e8f9906e6b8db49f358e7b52 \
-    --hash=sha256:0d48200bc50cbf498716712129eef819b1729339e34c3ae71656964dac907c28 \
+    --hash=sha256:133ba06bad4e5f2f8bf6a0ac434e0fd686df749a86b3478903b92ec3a9c0c90b \
-    --hash=sha256:23d12eacffa9d0f290c0fe0c4e81ba6d5f3a5b7ac3c30a5eaf0126bf4deda5c8 \
+    --hash=sha256:1429dc183b36ec972055e13250d96e174491559433eb3061691b446899b87384 \
-    --hash=sha256:37c9ba82bd82eb6a23c2e5acc03055c0e45697253b2393c9a50cef76a3985304 \
+    --hash=sha256:1b805231bfb7b2900a16638c3c8b45c694334c811f84463e52451e00c9412691 \
-    --hash=sha256:51503524dd6f152ab4ad1fbd168fc6c30b5795e8c70be4410a64940b3abb55c0 \
+    --hash=sha256:3a35e33902b2e6079949feed7a2dafa5ac6f019da97bd255842bb22de3c11bf5 \
-    --hash=sha256:8041e2de00e745c0e05a502d6e6db310db7faa7c979b3a5877123548a4c0b214 \
+    --hash=sha256:5ea034d040e6ab1d2ae04ab05a3f37dbd719c4dee3804b13903d4cc794b1336e \
-    --hash=sha256:81fcd96a275209ef117e9ec91f75c731fa18dcfd9ffaa1c0adbdaa3616a86043 \
+    --hash=sha256:682328aa576ec393c1872615bcb877cf32d800d4a2f150e1a5dc7e56644010b1 \
-    --hash=sha256:853da4f9563d982e4121fed8c92eea1a4594a2299037b3034c3c898cb8e933d6 \
+    --hash=sha256:6e06eac722676797e8fce4adb8ad3dc57a1bb3adfb0dd3fdf8306c055a38456c \
-    --hash=sha256:8b4572c334593d449113f9dc8d19b93b7b271bdbe90ba7509eb178923327b625 \
+    --hash=sha256:7eed31f4efc8356e200568ba05ad645525f1fbd8674f1e5be61a493e715e3873 \
-    --hash=sha256:9416443e219356e3c31f1f918a91badf2e37acf297e2fa13d24d1cc2380f8fbc \
+    --hash=sha256:80cb0380838bf4e48da6adedb0c7cd060c187bb4a75f67a5aa9ec33689b84872 \
-    --hash=sha256:9854f612e1b59ec66804931df5add3b2d5ef0067748ea29dc60f0efdcda9a638 \
+    --hash=sha256:b0b2a984bbfc543d144d88caad6cc7ff4a71be77102014bd617bd88cfb038727 \
-    --hash=sha256:99a26afdb82ea83a265137a398f570402aa1f2b5dfb4ac3300c026931817b163 \
+    --hash=sha256:c196a5394c56352e21cb7224739c6dd0075b69dd56f758505951d1d8d68cf8a9 \
-    --hash=sha256:a19bf883b3384957e4a4a13e6bd1ae3d85ae87f4beb5957e35b0be287f12f4e4 \
+    --hash=sha256:d83c1d38658b0f81c282b41238092ed89d8f93c6e342224ab73fb39e16848721 \
-    --hash=sha256:a9f145660588187ff835c55a7d2ddf6abfc570c2651c276d3d4be8a2766db490 \
+    --hash=sha256:df7de669cbf21de4b04a3ffc9920bc8426cab4c61365fa84d79bf97401a8bef7 \
-    --hash=sha256:ac57fcdcfb0b73bb3203b58a14501abb7e5ff9ea5e2edfa06bb03035f0cff248 \
+    --hash=sha256:e5db19d4a7d41bbeb3dd89b49fc1bc7e6e515b51bbf32589c618655a0ebe0bf0 \
-    --hash=sha256:bcb530089ff24f6458a81ac3fa699e8c00194208a724b644ecc68422e1111939 \
+    --hash=sha256:e695ac8c3efe124d998230b219eb51afb6ef10524a50b3c45109c4b77a8a3a92 \
-    --hash=sha256:beeabe25c3b704f7d56b573f7d2ff88fc99f0138e43480cecdfcaa3b87fe4f87 \
+    --hash=sha256:eac2a3f659d5f41d6bbfb6a97733bc7800ea5e906dc873732e00cebb98cec9e4 \
-    --hash=sha256:d634a7ea1fc3380ff96f9e44d8d22f38418c1c381d5fac680b272d7d90883720 \
-    --hash=sha256:d97b0661e1aead761f0ded3b769044bb00ed5d33e1ec865e891a8b128bf7c656 \
     # via gevent
 idna==2.7 \
     --hash=sha256:156a6814fb5ac1fc6850fb002e0852d56c0c8d2531923a51032d1b70760e186e \
@@ -146,9 +149,9 @@ markupsafe==1.1.1 \
     --hash=sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f \
     --hash=sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7 \
     # via jinja2
-mimcvdf==1.0.0 \
+mimcvdf==1.1.0 \
-    --hash=sha256:4675261d3a49c5665063fdaf256765d160617ca4301601aa1450e98bb96afbbc \
+    --hash=sha256:97a4ccdebb58352c64c268d2e57ef8817c9fe4ac3dcc922410bfcc72033f344a \
-    --hash=sha256:c0d4a79a52078c2ec13c6a5eaa086681a947669708a9b9a2e3b6cd6518c59371 \
+    --hash=sha256:ae47c79bfd6b7b76077c8ce3301a48a7c10a609d8a882e7bd785e2ef851ecd28 \
     # via -r requirements.in
 niceware==0.2.1 \
     --hash=sha256:0f8b192f2a1e800e068474f6e208be9c7e2857664b33a96f4045340de4e5c69c \
@@ -157,51 +160,38 @@ niceware==0.2.1 \
 pathtools==0.1.2 \
     --hash=sha256:7c35c5421a39bb82e58018febd90e3b6e5db34c5443aaaf742b3f33d4655f1c0 \
     # via watchdog
-psutil==5.7.0 \
+psutil==5.7.2 \
-    --hash=sha256:1413f4158eb50e110777c4f15d7c759521703bd6beb58926f1d562da40180058 \
+    --hash=sha256:0ee3c36428f160d2d8fce3c583a0353e848abb7de9732c50cf3356dd49ad63f8 \
-    --hash=sha256:298af2f14b635c3c7118fd9183843f4e73e681bb6f01e12284d4d70d48a60953 \
+    --hash=sha256:10512b46c95b02842c225f58fa00385c08fa00c68bac7da2d9a58ebe2c517498 \
-    --hash=sha256:60b86f327c198561f101a92be1995f9ae0399736b6eced8f24af41ec64fb88d4 \
+    --hash=sha256:4080869ed93cce662905b029a1770fe89c98787e543fa7347f075ade761b19d6 \
-    --hash=sha256:685ec16ca14d079455892f25bd124df26ff9137664af445563c1bd36629b5e0e \
+    --hash=sha256:5e9d0f26d4194479a13d5f4b3798260c20cecf9ac9a461e718eb59ea520a360c \
-    --hash=sha256:73f35ab66c6c7a9ce82ba44b1e9b1050be2a80cd4dcc3352cc108656b115c74f \
+    --hash=sha256:66c18ca7680a31bf16ee22b1d21b6397869dda8059dbdb57d9f27efa6615f195 \
-    --hash=sha256:75e22717d4dbc7ca529ec5063000b2b294fc9a367f9c9ede1f65846c7955fd38 \
+    --hash=sha256:68d36986ded5dac7c2dcd42f2682af1db80d4bce3faa126a6145c1637e1b559f \
-    --hash=sha256:a02f4ac50d4a23253b68233b07e7cdb567bd025b982d5cf0ee78296990c22d9e \
+    --hash=sha256:90990af1c3c67195c44c9a889184f84f5b2320dce3ee3acbd054e3ba0b4a7beb \
-    --hash=sha256:d008ddc00c6906ec80040d26dc2d3e3962109e40ad07fd8a12d0284ce5e0e4f8 \
+    --hash=sha256:a5b120bb3c0c71dfe27551f9da2f3209a8257a178ed6c628a819037a8df487f1 \
-    --hash=sha256:d84029b190c8a66a946e28b4d3934d2ca1528ec94764b180f7d6ea57b0e75e26 \
+    --hash=sha256:d8a82162f23c53b8525cf5f14a355f5d1eea86fa8edde27287dd3a98399e4fdf \
-    --hash=sha256:e2d0c5b07c6fe5a87fa27b7855017edb0d52ee73b71e6ee368fae268605cc3f5 \
+    --hash=sha256:f2018461733b23f308c298653c8903d32aaad7873d25e1d228765e91ae42c3f2 \
-    --hash=sha256:f344ca230dd8e8d5eee16827596f1c22ec0876127c28e800d7ae20ed44c4b310 \
+    --hash=sha256:ff1977ba1a5f71f89166d5145c3da1cea89a0fdb044075a12c720ee9123ec818 \
     # via -r requirements.in
 pycparser==2.19 \
     --hash=sha256:a988718abfad80b6b157acce7bf130a30876d27603738ac39f140993246b25b3 \
     # via cffi
-pynacl==1.3.0 \
+pynacl==1.4.0 \
-    --hash=sha256:05c26f93964373fc0abe332676cb6735f0ecad27711035b9472751faa8521255 \
+    --hash=sha256:30f9b96db44e09b3304f9ea95079b1b7316b2b4f3744fe3aaecccd95d547063d \
-    --hash=sha256:0c6100edd16fefd1557da078c7a31e7b7d7a52ce39fdca2bec29d4f7b6e7600c \
+    --hash=sha256:54e9a2c849c742006516ad56a88f5c74bf2ce92c9f67435187c3c5953b346505 \
-    --hash=sha256:0d0a8171a68edf51add1e73d2159c4bc19fc0718e79dec51166e940856c2f28e \
+    --hash=sha256:7757ae33dae81c300487591c68790dfb5145c7d03324000433d9a2c141f82af7 \
-    --hash=sha256:1c780712b206317a746ace34c209b8c29dbfd841dfbc02aa27f2084dd3db77ae \
+    --hash=sha256:95be540c5a3359019d999c91d6ff0b59c6c30f85a70b8860a885fdf1a3f92840 \
-    --hash=sha256:2424c8b9f41aa65bbdbd7a64e73a7450ebb4aa9ddedc6a081e7afcc4c97f7621 \
+    --hash=sha256:c878f2a3f9ccbce71167da0847797198138af35a7ae3f8673e80d7a6c5336d94 \
-    --hash=sha256:2d23c04e8d709444220557ae48ed01f3f1086439f12dbf11976e849a4926db56 \
+    --hash=sha256:d452a6746f0a7e11121e64625109bc4468fc3100452817001dbe018bb8b08514 \
-    --hash=sha256:30f36a9c70450c7878053fa1344aca0145fd47d845270b43a7ee9192a051bf39 \
-    --hash=sha256:37aa336a317209f1bb099ad177fef0da45be36a2aa664507c5d72015f956c310 \
-    --hash=sha256:4943decfc5b905748f0756fdd99d4f9498d7064815c4cf3643820c9028b711d1 \
-    --hash=sha256:57ef38a65056e7800859e5ba9e6091053cd06e1038983016effaffe0efcd594a \
-    --hash=sha256:5bd61e9b44c543016ce1f6aef48606280e45f892a928ca7068fba30021e9b786 \
-    --hash=sha256:6482d3017a0c0327a49dddc8bd1074cc730d45db2ccb09c3bac1f8f32d1eb61b \
-    --hash=sha256:7d3ce02c0784b7cbcc771a2da6ea51f87e8716004512493a2b69016326301c3b \
-    --hash=sha256:a14e499c0f5955dcc3991f785f3f8e2130ed504fa3a7f44009ff458ad6bdd17f \
-    --hash=sha256:a39f54ccbcd2757d1d63b0ec00a00980c0b382c62865b61a505163943624ab20 \
-    --hash=sha256:aabb0c5232910a20eec8563503c153a8e78bbf5459490c49ab31f6adf3f3a415 \
-    --hash=sha256:bd4ecb473a96ad0f90c20acba4f0bf0df91a4e03a1f4dd6a4bdc9ca75aa3a715 \
-    --hash=sha256:e2da3c13307eac601f3de04887624939aca8ee3c9488a0bb0eca4fb9401fc6b1 \
-    --hash=sha256:f67814c38162f4deb31f68d590771a29d5ae3b1bd64b75cf232308e5c74777e0 \
     # via -r requirements.in
 pysocks==1.7.1 \
     --hash=sha256:08e69f092cc6dbe92a0fdd16eeb9b9ffbc13cadfe5ca4c7bd92ffb078b293299 \
     --hash=sha256:2725bd0a9925919b9b51739eea5f9e2bae91e83288108a9ad338b2e3a4435ee5 \
     --hash=sha256:3f8804571ebe159c380ac6de37643bb4685970655d3bba243530d6558b799aa0 \
     # via -r requirements.in
-requests==2.23.0 \
+requests==2.24.0 \
-    --hash=sha256:43999036bfa82904b6af1d99e4882b560e5e2c68e5c4b0aa03b655f3d7d73fee \
+    --hash=sha256:b3559a131db72c33ee969480840fff4bb6dd111de7dd27c8ee1f820f4f00231b \
-    --hash=sha256:b3f43d496c6daba4493e7c431722aeb7dbc6288f52a6e04e7b6023b0247817e6 \
+    --hash=sha256:fe75cc94a9443b9246fc7049224f75604b113c36acb93f87b80ed42c44cbb898 \
     # via -r requirements.in, streamedrequests
 six==1.12.0 \
     --hash=sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c \
@@ -210,24 +200,90 @@ six==1.12.0 \
 stem==1.8.0 \
     --hash=sha256:a0b48ea6224e95f22aa34c0bc3415f0eb4667ddeae3dfb5e32a6920c185568c2 \
     # via -r requirements.in
-streamedrequests==1.0.0 \
+streamedrequests==1.0.3 \
-    --hash=sha256:1d9d07394804a6e1fd66bde74a804e71cab98e6920053865574a459f1cf7d3b7 \
+    --hash=sha256:4388ffc0ee94dda719dafc4324b8ddd108cb2231ec59871de79e2592bf4eef0a \
+    --hash=sha256:ee68417a1522e75c35b1b2d5f3b6f7e76a3a1a6c0ef5e0c573d08307910079d8 \
     # via -r requirements.in
 toomanyobjs==1.1.0 \
     --hash=sha256:99e27468f9dad19127be9e2fb086b42acd69aed9ad7e63cef74d6e4389be0534 \
     # via -r requirements.in
+ujson==3.0.0 \
+    --hash=sha256:019a17e7162f26e264f1645bb41630f7103d178c092ea4bb8f3b16126c3ea210 \
+    --hash=sha256:0379ffc7484b862a292e924c15ad5f1c5306d4271e2efd162144812afb08ff97 \
+    --hash=sha256:0959a5b569e192459b492b007e3fd63d8f4b4bcb4f69dcddca850a9b9dfe3e7a \
+    --hash=sha256:0e2352b60c4ac4fc75b723435faf36ef5e7f3bfb988adb4d589b5e0e6e1d90aa \
+    --hash=sha256:0f33359908df32033195bfdd59ba2bfb90a23cb280ef9a0ba11e5013a53d7fd9 \
+    --hash=sha256:154f778f0b028390067aaedce8399730d4f528a16a1c214fe4eeb9c4e4f51810 \
+    --hash=sha256:3bd791d17a175c1c6566aeaec1755b58e3f021fe9bb62f10f02b656b299199f5 \
+    --hash=sha256:634c206f4fb3be7e4523768c636d2dd41cb9c7130e2d219ef8305b8fb6f4838e \
+    --hash=sha256:670018d4ab4b0755a7234a9f4791723abcd0506c0eed33b2ed50579c4aff31f2 \
+    --hash=sha256:9c68557da3e3ad57e0105aceba0cce5f8f7cd07d207c3860e59c0b3044532830 \
+    --hash=sha256:a32f2def62b10e8a19084d17d40363c4da1ac5f52d300a9e99d7efb49fe5f34a \
+    --hash=sha256:bea2958c7b5bf4f191f0def751b6f7c8b208edb5f7277e21776329f2ca042385 \
+    --hash=sha256:c04d253fec814657fd9f150ef2333dbd0bc6f46208355aa753a29e0696b7fa7e \
+    --hash=sha256:c841a6450d64c24c64cbcca429bab22cdb6daef5eaddfdfebe798a5e9e5aff4c \
+    --hash=sha256:e0199849d61cc6418f94d52a314c6a27524d65e82174d2a043fb718f73d1520d \
+    --hash=sha256:f40bb0d0cb534aad3e24884cf864bda7a71eb5984bd1da61d1711bbfb3be2c38 \
+    --hash=sha256:f854702a9aff3a445f4a0b715d240f2a3d84014d8ae8aad05a982c7ffab12525 \
+    # via -r requirements.in
 unpaddedbase32==0.2.0 \
     --hash=sha256:4aacee75f8fd6c8cf129842ecba45ca59c11bfb13dae19d86f32b48fa3715403 \
     --hash=sha256:b7b780c31d27d55e66abf6c221216a35690ee8892c2daacff7f2528e229bd9c3 \
     # via -r requirements.in
-urllib3==1.25.8 \
+urllib3==1.25.9 \
-    --hash=sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc \
+    --hash=sha256:3018294ebefce6572a474f0604c2021e33b3fd8006ecd11d62107a5d2a963527 \
-    --hash=sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc \
+    --hash=sha256:88206b0eb87e6d677d424843ac5209e3fb9d0190d0ee169599165ec25e9d9115 \
     # via -r requirements.in, requests
-watchdog==0.10.2 \
+watchdog==0.10.3 \
-    --hash=sha256:c560efb643faed5ef28784b2245cf8874f939569717a4a12826a173ac644456b \
+    --hash=sha256:4214e1379d128b0588021880ccaf40317ee156d4603ac388b9adcf29165e0c04 \
     # via -r requirements.in
 werkzeug==0.15.5 \
     --hash=sha256:87ae4e5b5366da2347eb3116c0e6c681a0e939a33b2805e2c0cbd282664932c4 \
     --hash=sha256:a13b74dd3c45f758d4ebdb224be8f1ab8ef58b3c0ffc1783a8c7d9f4f50227e6 \
     # via flask
+zope.event==4.4 \
+    --hash=sha256:69c27debad9bdacd9ce9b735dad382142281ac770c4a432b533d6d65c4614bcf \
+    --hash=sha256:d8e97d165fd5a0997b45f5303ae11ea3338becfe68c401dd88ffd2113fe5cae7 \
+    # via gevent
+zope.interface==5.1.0 \
+    --hash=sha256:0103cba5ed09f27d2e3de7e48bb320338592e2fabc5ce1432cf33808eb2dfd8b \
+    --hash=sha256:14415d6979356629f1c386c8c4249b4d0082f2ea7f75871ebad2e29584bd16c5 \
+    --hash=sha256:1ae4693ccee94c6e0c88a4568fb3b34af8871c60f5ba30cf9f94977ed0e53ddd \
+    --hash=sha256:1b87ed2dc05cb835138f6a6e3595593fea3564d712cb2eb2de963a41fd35758c \
+    --hash=sha256:269b27f60bcf45438e8683269f8ecd1235fa13e5411de93dae3b9ee4fe7f7bc7 \
+    --hash=sha256:27d287e61639d692563d9dab76bafe071fbeb26818dd6a32a0022f3f7ca884b5 \
+    --hash=sha256:39106649c3082972106f930766ae23d1464a73b7d30b3698c986f74bf1256a34 \
+    --hash=sha256:40e4c42bd27ed3c11b2c983fecfb03356fae1209de10686d03c02c8696a1d90e \
+    --hash=sha256:461d4339b3b8f3335d7e2c90ce335eb275488c587b61aca4b305196dde2ff086 \
+    --hash=sha256:4f98f70328bc788c86a6a1a8a14b0ea979f81ae6015dd6c72978f1feff70ecda \
+    --hash=sha256:558a20a0845d1a5dc6ff87cd0f63d7dac982d7c3be05d2ffb6322a87c17fa286 \
+    --hash=sha256:562dccd37acec149458c1791da459f130c6cf8902c94c93b8d47c6337b9fb826 \
+    --hash=sha256:5e86c66a6dea8ab6152e83b0facc856dc4d435fe0f872f01d66ce0a2131b7f1d \
+    --hash=sha256:60a207efcd8c11d6bbeb7862e33418fba4e4ad79846d88d160d7231fcb42a5ee \
+    --hash=sha256:645a7092b77fdbc3f68d3cc98f9d3e71510e419f54019d6e282328c0dd140dcd \
+    --hash=sha256:6874367586c020705a44eecdad5d6b587c64b892e34305bb6ed87c9bbe22a5e9 \
+    --hash=sha256:74bf0a4f9091131de09286f9a605db449840e313753949fe07c8d0fe7659ad1e \
+    --hash=sha256:7b726194f938791a6691c7592c8b9e805fc6d1b9632a833b9c0640828cd49cbc \
+    --hash=sha256:8149ded7f90154fdc1a40e0c8975df58041a6f693b8f7edcd9348484e9dc17fe \
+    --hash=sha256:8cccf7057c7d19064a9e27660f5aec4e5c4001ffcf653a47531bde19b5aa2a8a \
+    --hash=sha256:911714b08b63d155f9c948da2b5534b223a1a4fc50bb67139ab68b277c938578 \
+    --hash=sha256:a5f8f85986197d1dd6444763c4a15c991bfed86d835a1f6f7d476f7198d5f56a \
+    --hash=sha256:a744132d0abaa854d1aad50ba9bc64e79c6f835b3e92521db4235a1991176813 \
+    --hash=sha256:af2c14efc0bb0e91af63d00080ccc067866fb8cbbaca2b0438ab4105f5e0f08d \
+    --hash=sha256:b054eb0a8aa712c8e9030065a59b5e6a5cf0746ecdb5f087cca5ec7685690c19 \
+    --hash=sha256:b0becb75418f8a130e9d465e718316cd17c7a8acce6fe8fe07adc72762bee425 \
+    --hash=sha256:b1d2ed1cbda2ae107283befd9284e650d840f8f7568cb9060b5466d25dc48975 \
+    --hash=sha256:ba4261c8ad00b49d48bbb3b5af388bb7576edfc0ca50a49c11dcb77caa1d897e \
+    --hash=sha256:d1fe9d7d09bb07228650903d6a9dc48ea649e3b8c69b1d263419cc722b3938e8 \
+    --hash=sha256:d7804f6a71fc2dda888ef2de266727ec2f3915373d5a785ed4ddc603bbc91e08 \
+    --hash=sha256:da2844fba024dd58eaa712561da47dcd1e7ad544a257482392472eae1c86d5e5 \
+    --hash=sha256:dcefc97d1daf8d55199420e9162ab584ed0893a109f45e438b9794ced44c9fd0 \
+    --hash=sha256:dd98c436a1fc56f48c70882cc243df89ad036210d871c7427dc164b31500dc11 \
+    --hash=sha256:e74671e43ed4569fbd7989e5eecc7d06dc134b571872ab1d5a88f4a123814e9f \
+    --hash=sha256:eb9b92f456ff3ec746cd4935b73c1117538d6124b8617bc0fe6fda0b3816e345 \
+    --hash=sha256:ebb4e637a1fb861c34e48a00d03cffa9234f42bef923aec44e5625ffb9a8e8f9 \
+    --hash=sha256:ef739fe89e7f43fb6494a43b1878a36273e5924869ba1d866f752c5812ae8d58 \
+    --hash=sha256:f40db0e02a8157d2b90857c24d89b6310f9b6c3642369852cdc3b5ac49b92afc \
+    --hash=sha256:f68bf937f113b88c866d090fea0bc52a098695173fc613b055a17ff0cf9683b6 \
+    --hash=sha256:fb55c182a3f7b84c1a2d6de5fa7b1a05d4660d866b91dbf8d74549c57a1499e8 \
+    # via gevent

scripts/disable-dev-config.py

@@ -26,6 +26,7 @@ conf['transports']['tor'] = True
 conf['transports']['sneakernet'] = True
 conf['statistics']['i_dont_want_privacy'] = False
 conf['statistics']['server'] = ''
+conf['ui']['animated_background'] = True
 
 json.dump(conf, open('static-data/default_config.json', 'w'), sort_keys=True, indent=4)
 

scripts/enable-dev-config.py

@@ -27,6 +27,7 @@ conf['onboarding']['done'] = True
 conf['general']['minimum_block_pow'] = block_pow
 conf['general']['minimum_send_pow'] = block_pow
 conf['log']['file']['remove_on_exit'] = False
+conf['ui']['animated_background'] = False
 if input('Stat reporting? y/n') == 'y':
     conf['statistics']['i_dont_want_privacy'] = True
     conf['statistics']['server'] = input('Statistics server')

src/__init__.py

@@ -5,6 +5,13 @@ This file initializes Onionr when ran to be a daemon or with commands
 
 Run with 'help' for usage.
 """
+import sys
+try:
+    import sqlite3
+except ModuleNotFoundError:
+    sys.stderr.write(
+        'Error, Onionr requires Sqlite3-enabled Python.\n')
+    sys.exit(1)
 """
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -28,12 +35,12 @@ ran_as_script = False
 if __name__ == "__main__": ran_as_script = True
 
 # Import standard libraries
-import sys  # noqa
 
 try:
     from etc import dependencycheck  # noqa
 except ModuleNotFoundError as e:
-    print('Onionr needs ' + str(e) + ' installed')
+    print('Missing requirement: ' + str(e) + ' installed')
+    sys.exit(1)
 
 # Import 3rd party libraries
 

src/bigbrother/ministry/ofdisk.py

@@ -9,7 +9,6 @@ def detect_disk_access(info):
 
     whitelist = [identify_home(), 'onionr/src/', '/site-packages/', '/usr/lib64/']
 
-
     for item in whitelist:
         if item in info[0]:
             return

src/communicator/__init__.py

@@ -26,6 +26,7 @@ from communicatorutils import housekeeping
 from communicatorutils import netcheck
 from onionrutils import localcommand
 from onionrutils import epoch
+from onionrcommands.openwebinterface import get_url
 from etc import humanreadabletime
 import onionrservices
 import filepaths
@@ -241,6 +242,8 @@ class OnionrCommunicatorDaemon:
             logger.info(
                 'First run detected. Run openhome to get setup.',
                 terminal=True)
+            get_url()
+            
 
             while not config.get('onboarding.done', True) and \
                     not self.shutdown:

src/communicator/onlinepeers/onlinepeers.py

@@ -55,6 +55,10 @@ def get_online_peers(comm_inst: 'OnionrCommunicatorDaemon'):
         if len(comm_inst.onlinePeers) == 0:
             logger.debug('Couldn\'t connect to any peers.' +
                          f' Last node seen {last_seen}  ago.')
+            try:
+                get_online_peers(comm_inst)
+            except RecursionError:
+                pass
         else:
             comm_inst.lastNodeSeen = time.time()
     comm_inst.decrementThreadCount('get_online_peers')

src/communicatorutils/announcenode.py

@@ -1,14 +1,11 @@
 """
-    Onionr - Private P2P Communication.
+Onionr - Private P2P Communication.
 
 Use a communicator instance to announce
 our transport address to connected nodes
 """
-import base64
-import onionrproofs
 import logger
-from etc import onionrvalues
+from onionrutils import basicrequests
-from onionrutils import basicrequests, bytesconverter
 from utils import gettransports
 from netcontroller import NetController
 from communicator import onlinepeers
@@ -33,7 +30,6 @@ import onionrexceptions
 def announce_node(daemon):
     """Announce our node to our peers."""
     ret_data = False
-    announce_fail = False
 
     # Do not let announceCache get too large
     if len(daemon.announceCache) >= 10000:

src/communicatorutils/connectnewpeers.py

@@ -1,9 +1,19 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Connect a new peer to our communicator instance. Does so randomly if no peer is specified
+Connect a new peer to our communicator instance.
-'''
+Does so randomly if no peer is specified
-'''
+"""
+import time
+import secrets
+
+import onionrexceptions
+import logger
+import onionrpeers
+from utils import networkmerger, gettransports
+from onionrutils import stringvalidators, epoch
+from communicator import peeraction, bootstrappeers
+from coredb import keydb
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,13 +26,9 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import time, sys, secrets
+
-import onionrexceptions, logger, onionrpeers
+
-from utils import networkmerger, gettransports
-from onionrutils import stringvalidators, epoch
-from communicator import peeraction, bootstrappeers
-from coredb import keydb
 def connect_new_peer_to_communicator(comm_inst, peer='', useBootstrap=False):
     config = comm_inst.config
     retData = False
@@ -32,14 +38,18 @@ def connect_new_peer_to_communicator(comm_inst, peer='', useBootstrap=False):
         if stringvalidators.validate_transport(peer):
             peerList = [peer]
         else:
-            raise onionrexceptions.InvalidAddress('Will not attempt connection test to invalid address')
+            raise onionrexceptions.InvalidAddress(
+                'Will not attempt connection test to invalid address')
     else:
         peerList = keydb.listkeys.list_adders()
 
     mainPeerList = keydb.listkeys.list_adders()
     peerList = onionrpeers.get_score_sorted_peer_list()
 
-    # If we don't have enough peers connected or random chance, select new peers to try
+    """
+    If we don't have enough peers connected or random chance,
+    select new peers to try
+    """
     if len(peerList) < 8 or secrets.randbelow(4) == 3:
         tryingNew = []
         for x in comm_inst.newPeers:
@@ -60,8 +70,12 @@ def connect_new_peer_to_communicator(comm_inst, peer='', useBootstrap=False):
         # Don't connect to our own address
         if address in transports:
             continue
-        # Don't connect to invalid address or if its already been tried/connected, or if its cooled down
+        """Don't connect to invalid address or
-        if len(address) == 0 or address in tried or address in comm_inst.onlinePeers or address in comm_inst.cooldownPeer:
+        if its already been tried/connected, or if its cooled down
+        """
+        if len(address) == 0 or address in tried \
+            or address in comm_inst.onlinePeers \
+                or address in comm_inst.cooldownPeer:
             continue
         if comm_inst.shutdown:
             return
@@ -70,7 +84,7 @@ def connect_new_peer_to_communicator(comm_inst, peer='', useBootstrap=False):
         if ret == 'pong!':
             time.sleep(0.1)
             if address not in mainPeerList:
-                # Add a peer to our list if it isn't already since it successfully connected
+                # Add a peer to our list if it isn't already since it connected
                 networkmerger.mergeAdders(address)
             if address not in comm_inst.onlinePeers:
                 logger.info('Connected to ' + address, terminal=True)
@@ -83,7 +97,8 @@ def connect_new_peer_to_communicator(comm_inst, peer='', useBootstrap=False):
                 if profile.address == address:
                     break
             else:
-                comm_inst.peerProfiles.append(onionrpeers.PeerProfiles(address))
+                comm_inst.peerProfiles.append(
+                    onionrpeers.PeerProfiles(address))
             break
         else:
             # Mark a peer as tried if they failed to respond to ping

src/communicatorutils/cooldownpeer.py

@@ -1,9 +1,10 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Select a random online peer in a communicator instance and have them "cool down"
+Select random online peer in a communicator instance and have them "cool down"
-'''
+"""
-'''
+from onionrutils import epoch
+from communicator import onlinepeers
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,39 +17,39 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-from onionrutils import epoch
+
-from communicator import onlinepeers
+
 def cooldown_peer(comm_inst):
-    '''Randomly add an online peer to cooldown, so we can connect a new one'''
+    """Randomly add an online peer to cooldown, so we can connect a new one."""
     config = comm_inst.config
-    onlinePeerAmount = len(comm_inst.onlinePeers)
+    online_peer_amount = len(comm_inst.onlinePeers)
     minTime = 300
-    cooldownTime = 600
+    cooldown_time = 600
-    toCool = ''
+    to_cool = ''
     tempConnectTimes = dict(comm_inst.connectTimes)
 
     # Remove peers from cooldown that have been there long enough
     tempCooldown = dict(comm_inst.cooldownPeer)
     for peer in tempCooldown:
-        if (epoch.get_epoch() - tempCooldown[peer]) >= cooldownTime:
+        if (epoch.get_epoch() - tempCooldown[peer]) >= cooldown_time:
             del comm_inst.cooldownPeer[peer]
 
     # Cool down a peer, if we have max connections alive for long enough
-    if onlinePeerAmount >= config.get('peers.max_connect', 10, save = True):
+    if online_peer_amount >= config.get('peers.max_connect', 10, save=True):
         finding = True
 
         while finding:
             try:
-                toCool = min(tempConnectTimes, key=tempConnectTimes.get)
+                to_cool = min(tempConnectTimes, key=tempConnectTimes.get)
-                if (epoch.get_epoch() - tempConnectTimes[toCool]) < minTime:
+                if (epoch.get_epoch() - tempConnectTimes[to_cool]) < minTime:
-                    del tempConnectTimes[toCool]
+                    del tempConnectTimes[to_cool]
                 else:
                     finding = False
             except ValueError:
                 break
         else:
-            onlinepeers.remove_online_peer(comm_inst, toCool)
+            onlinepeers.remove_online_peer(comm_inst, to_cool)
-            comm_inst.cooldownPeer[toCool] = epoch.get_epoch()
+            comm_inst.cooldownPeer[to_cool] = epoch.get_epoch()
 
-    comm_inst.decrementThreadCount('cooldown_peer')
+    comm_inst.decrementThreadCount('cooldown_peer')

src/communicatorutils/deniableinserts.py

@@ -1,9 +1,12 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Use the communicator to insert fake mail messages
+Use the communicator to insert fake mail messages
-'''
+"""
-'''
+import secrets
+
+from etc import onionrvalues
+import onionrblocks
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,17 +19,18 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import secrets
+
-from etc import onionrvalues
+
-import onionrblocks
 def insert_deniable_block(comm_inst):
-    '''Insert a fake block in order to make it more difficult to track real blocks'''
+    """Insert a fake block to make it more difficult to track real blocks."""
     fakePeer = ''
     chance = 10
     if secrets.randbelow(chance) == (chance - 1):
         # This assumes on the libsodium primitives to have key-privacy
         fakePeer = onionrvalues.DENIABLE_PEER_ADDRESS
         data = secrets.token_hex(secrets.randbelow(5120) + 1)
-        onionrblocks.insert(data, header='pm', encryptType='asym', asymPeer=fakePeer, disableForward=True, meta={'subject': 'foo'})
+        onionrblocks.insert(data, header='pm', encryptType='asym',
-    comm_inst.decrementThreadCount('insert_deniable_block')
+                            asymPeer=fakePeer, disableForward=True,
+                            meta={'subject': 'foo'})
+    comm_inst.decrementThreadCount('insert_deniable_block')

src/communicatorutils/downloadblocks/__init__.py

@@ -126,7 +126,7 @@ def download_blocks_from_communicator(comm_inst: "OnionrCommunicatorDaemon"):
                                 f'/daemon-event/upload_event',
                                 post=True,
                                 is_json=True,
-                                postData={'block': blockHash}
+                                post_data={'block': blockHash}
                             )
                     else:
                         logger.warn('POW failed for block %s.' % (blockHash,))

src/communicatorutils/housekeeping.py

@@ -4,6 +4,7 @@ Cleanup old Onionr blocks and forward secrecy keys using the communicator.
 Ran from a communicator timer usually
 """
 import sqlite3
+
 import logger
 from onionrusers import onionrusers
 from onionrutils import epoch
@@ -67,7 +68,8 @@ def clean_keys(comm_inst):
     time = epoch.get_epoch()
     deleteKeys = []
 
-    for entry in c.execute("SELECT * FROM forwardKeys WHERE expire <= ?", (time,)):
+    for entry in c.execute(
+            "SELECT * FROM forwardKeys WHERE expire <= ?", (time,)):
         logger.debug('Forward key: %s' % entry[1])
         deleteKeys.append(entry[1])
 

src/communicatorutils/lookupadders.py

@@ -1,9 +1,14 @@
-'''
+"""
-    Onionr - Private P2P Communication
+Onionr - Private P2P Communication.
 
-    Lookup new peer transport addresses using the communicator
+Lookup new peer transport addresses using the communicator
-'''
+"""
-'''
+import logger
+from onionrutils import stringvalidators
+from communicator import peeraction, onlinepeers
+from utils import gettransports
+import onionrexceptions
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,12 +21,9 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import logger
+
-from onionrutils import stringvalidators
+
-from communicator import peeraction, onlinepeers
-from utils import gettransports
-import onionrexceptions
 def lookup_new_peer_transports_with_communicator(comm_inst):
     logger.info('Looking up new addresses...')
     tryAmount = 1
@@ -47,7 +49,8 @@ def lookup_new_peer_transports_with_communicator(comm_inst):
         invalid = []
         for x in newPeers:
             x = x.strip()
-            if not stringvalidators.validate_transport(x) or x in comm_inst.newPeers or x in transports:
+            if not stringvalidators.validate_transport(x) \
+                    or x in comm_inst.newPeers or x in transports:
                 # avoid adding if its our address
                 invalid.append(x)
         for x in invalid:
@@ -56,4 +59,5 @@ def lookup_new_peer_transports_with_communicator(comm_inst):
             except ValueError:
                 pass
         comm_inst.newPeers.extend(newPeers)
-    comm_inst.decrementThreadCount('lookup_new_peer_transports_with_communicator')
+    comm_inst.decrementThreadCount(
+        'lookup_new_peer_transports_with_communicator')

src/communicatorutils/lookupblocks.py

@@ -1,4 +1,4 @@
-"""Onionr - Private P2P Communication
+"""Onionr - Private P2P Communication.
 
 Lookup new blocks with the communicator using a random connected peer
 """
@@ -8,7 +8,7 @@ import logger
 import onionrproofs
 from onionrutils import stringvalidators, epoch
 from communicator import peeraction, onlinepeers
-from coredb import blockmetadb
+from coredb.blockmetadb import get_block_list
 from utils import reconstructhash
 from onionrblocks import onionrblacklist
 import onionrexceptions
@@ -36,20 +36,24 @@ def lookup_blocks_from_communicator(comm_inst):
     logger.info('Looking up new blocks')
     tryAmount = 2
     newBlocks = ''
-    existingBlocks = blockmetadb.get_block_list() # List of existing saved blocks
+    # List of existing saved blocks
-    triedPeers = [] # list of peers we've tried this time around
+    existingBlocks = get_block_list()
-    maxBacklog = 1560 # Max amount of *new* block hashes to have already in queue, to avoid memory exhaustion
+    triedPeers = []  # list of peers we've tried this time around
-    lastLookupTime = 0 # Last time we looked up a particular peer's list
+    # Max amount of *new* block hashes to have in queue
+    maxBacklog = 1560
+    lastLookupTime = 0  # Last time we looked up a particular peer's list
     new_block_count = 0
     for i in range(tryAmount):
-        listLookupCommand = 'getblocklist' # This is defined here to reset it each time
+        # Defined here to reset it each time, time offset is added later
+        listLookupCommand = 'getblocklist'
         if len(comm_inst.blockQueue) >= maxBacklog:
             break
         if not comm_inst.isOnline:
             break
         # check if disk allocation is used
         if comm_inst.storage_counter.is_full():
-            logger.debug('Not looking up new blocks due to maximum amount of allowed disk space used')
+            logger.debug(
+                'Not looking up new blocks due to maximum amount of disk used')
             break
         try:
             # select random online peer
@@ -65,33 +69,44 @@ def lookup_blocks_from_communicator(comm_inst):
                 continue
         triedPeers.append(peer)
 
-        # Get the last time we looked up a peer's stamp to only fetch blocks since then.
+        # Get the last time we looked up a peer's stamp,
+        # to only fetch blocks since then.
         # Saved in memory only for privacy reasons
         try:
             lastLookupTime = comm_inst.dbTimestamps[peer]
         except KeyError:
-            lastLookupTime = epoch.get_epoch() - config.get("general.max_block_age", onionrvalues.DEFAULT_EXPIRE)
+            lastLookupTime = epoch.get_epoch() - \
+                config.get("general.max_block_age",
+                           onionrvalues.DEFAULT_EXPIRE)
         listLookupCommand += '?date=%s' % (lastLookupTime,)
         try:
-            newBlocks = peeraction.peer_action(comm_inst, peer, listLookupCommand) # get list of new block hashes
+            newBlocks = peeraction.peer_action(
+                comm_inst,
+                peer, listLookupCommand)  # get list of new block hashes
         except Exception as error:
-            logger.warn('Could not get new blocks from %s.' % peer, error = error)
+            logger.warn(
+                f'Could not get new blocks from {peer}.',
+                error=error)
             newBlocks = False
-        else:
+
-            comm_inst.dbTimestamps[peer] = epoch.get_rounded_epoch(roundS=60)
+        if newBlocks != False:  # noqa
-        if newBlocks != False:
             # if request was a success
             for i in newBlocks.split('\n'):
                 if stringvalidators.validate_hash(i):
                     i = reconstructhash.reconstruct_hash(i)
                     # if newline seperated string is valid hash
-                    if not i in existingBlocks:
+
-                        # if block does not exist on disk and is not already in block queue
+                    # if block does not exist on disk + is not already in queue
+                    if i not in existingBlocks:
                         if i not in comm_inst.blockQueue:
-                            if onionrproofs.hashMeetsDifficulty(i) and not blacklist.inBlacklist(i):
+                            if onionrproofs.hashMeetsDifficulty(i) and \
+                                 not blacklist.inBlacklist(i):
                                 if len(comm_inst.blockQueue) <= 1000000:
-                                    comm_inst.blockQueue[i] = [peer] # add blocks to download queue
+                                    # add blocks to download queue
+                                    comm_inst.blockQueue[i] = [peer]
                                     new_block_count += 1
+                                    comm_inst.dbTimestamps[peer] = \
+                                        epoch.get_rounded_epoch(roundS=60)
                         else:
                             if peer not in comm_inst.blockQueue[i]:
                                 if len(comm_inst.blockQueue[i]) < 10:
@@ -100,7 +115,9 @@ def lookup_blocks_from_communicator(comm_inst):
         block_string = ""
         if new_block_count > 1:
             block_string = "s"
-        logger.info('Discovered %s new block%s' % (new_block_count, block_string), terminal=True)
+        logger.info(
-        comm_inst.download_blocks_timer.count = int(comm_inst.download_blocks_timer.frequency * 0.99)
+            f'Discovered {new_block_count} new block{block_string}',
+            terminal=True)
+        comm_inst.download_blocks_timer.count = \
+            int(comm_inst.download_blocks_timer.frequency * 0.99)
     comm_inst.decrementThreadCount('lookup_blocks_from_communicator')
-    return

src/communicatorutils/netcheck.py

@@ -1,9 +1,9 @@
 """
-    Onionr - Private P2P Communication
+Onionr - Private P2P Communication.
 
-    Determine if our node is able to use Tor based
+Determine if our node is able to use Tor based
-    on the status of a communicator instance
+on the status of a communicator instance
-    and the result of pinging onion http servers
+and the result of pinging onion http servers
 """
 import logger
 from utils import netutils
@@ -26,8 +26,10 @@ from . import restarttor
 
 
 def net_check(comm_inst):
-    """Check if we are connected to the internet
+    """Check if we are connected to the internet.
-    or not when we can't connect to any peers"""
+
+    or not when we can't connect to any peers
+    """
     # for detecting if we have received incoming connections recently
     rec = False
     if len(comm_inst.onlinePeers) == 0:
@@ -42,7 +44,8 @@ def net_check(comm_inst):
             if not comm_inst.shutdown:
                 if not comm_inst.config.get('general.offline_mode', False):
                     logger.warn('Network check failed, are you connected to ' +
-                                'the Internet, and is Tor working?',
+                                'the Internet, and is Tor working? ' +
+                                'This is usually temporary, but bugs and censorship can cause this to persist, in which case you should report it to beardog [at] mailbox.org',  # noqa
                                 terminal=True)
                     restarttor.restart(comm_inst)
                     comm_inst.offlinePeers = []

src/communicatorutils/onionrcommunicatortimers.py

@@ -1,10 +1,20 @@
-'''
+"""
-    Onionr - Private P2P Communication
+Onionr - Private P2P Communication.
 
-    This file contains timer control for the communicator
+This file contains timer control for the communicator
-'''
+"""
 from __future__ import annotations  # thank you python, very cool
-'''
+import uuid
+import threading
+
+import onionrexceptions
+import logger
+
+from typing import TYPE_CHECKING
+from typing import Callable, NewType, Iterable
+if TYPE_CHECKING:
+    from communicator import OnionrCommunicatorDaemon
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -17,26 +27,18 @@ from __future__ import annotations  # thank you python, very cool
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
 
-import uuid
-import threading
-
-import onionrexceptions, logger
-
-from typing import TYPE_CHECKING
-from typing import Callable, NewType, Iterable
-from psutil import Process
-if TYPE_CHECKING:
-    from communicator import OnionrCommunicatorDaemon
 
 CallFreqSeconds = NewType('CallFreqSeconds', int)
 
+
 class OnionrCommunicatorTimers:
     def __init__(self, daemon_inst: OnionrCommunicatorDaemon,
-                timer_function: Callable, frequency: CallFreqSeconds,
+                 timer_function: Callable, frequency: CallFreqSeconds,
-                make_thread:bool=True, thread_amount:int=1, max_threads:int=5,
+                 make_thread: bool = True,
-                requires_peer:bool=False, my_args:Iterable=[]):
+                 thread_amount: int = 1, max_threads: int = 5,
+                 requires_peer: bool = False, my_args: Iterable = []):
         self.timer_function = timer_function
         self.frequency = frequency
         self.thread_amount = thread_amount
@@ -51,30 +53,44 @@ class OnionrCommunicatorTimers:
 
     def processTimer(self):
 
-        # mark how many instances of a thread we have (decremented at thread end)
+        # mark # of instances of a thread we have (decremented at thread end)
         try:
             self.daemon_inst.threadCounts[self.timer_function.__name__]
         except KeyError:
             self.daemon_inst.threadCounts[self.timer_function.__name__] = 0
 
-        # execute thread if it is time, and we are not missing *required* online peer
+        # execute timer's func, if we are not missing *required* online peer
         if self.count == self.frequency and not self.daemon_inst.shutdown:
             try:
-                if self.requires_peer and len(self.daemon_inst.onlinePeers) == 0:
+                if self.requires_peer and \
+                        len(self.daemon_inst.onlinePeers) == 0:
                     raise onionrexceptions.OnlinePeerNeeded
             except onionrexceptions.OnlinePeerNeeded:
                 return
             else:
                 if self.make_thread:
                     for i in range(self.thread_amount):
-                        if self.daemon_inst.threadCounts[self.timer_function.__name__] >= self.max_threads:
+                        """
-                            logger.debug('%s is currently using the maximum number of threads, not starting another.' % self.timer_function.__name__)
+                        Log if a timer has max num of active threads
+                        If this logs frequently it is indicative of a bug
+                        or need for optimization
+                        """
+                        if self.daemon_inst.threadCounts[
+                                self.timer_function.__name__] >= \
+                                self.max_threads:
+                            logger.debug(
+                                f'{self.timer_function.__name__} is currently using the maximum number of threads, not starting another.')  # noqa
+                        # if active number of threads for timer not reached yet
                         else:
-                            self.daemon_inst.threadCounts[self.timer_function.__name__] += 1
+                            self.daemon_inst.threadCounts[
-                            newThread = threading.Thread(target=self.timer_function, args=self.args, daemon=True,
+                                self.timer_function.__name__] += 1
-                                                        name=self.timer_function.__name__ + ' - ' + str(uuid.uuid4()))
+                            newThread = threading.Thread(
+                                target=self.timer_function, args=self.args,
+                                daemon=True,
+                                name=self.timer_function.__name__ + ' - ' +
+                                str(uuid.uuid4()))
                             newThread.start()
                 else:
                     self.timer_function()
-            self.count = -1 # negative 1 because its incremented at bottom
+            self.count = -1  # negative 1 because its incremented at bottom
         self.count += 1

src/communicatorutils/proxypicker.py

@@ -1,9 +1,9 @@
-'''
+"""
-    Onionr - Private P2P Communication
+Onionr - Private P2P Communication.
 
-    Just picks a proxy to use based on a peer's address
+Pick a proxy to use based on a peer's address
-'''
+"""
-'''
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,11 +16,13 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
+
 
 def pick_proxy(peer_address):
     if peer_address.endswith('.onion'):
         return 'tor'
     elif peer_address.endswith('.i2p'):
         return 'i2p'
-    raise ValueError(f"Peer address was not string ending with acceptable value: {peer_address}")
+    raise ValueError(
+        f"Peer address not ending with acceptable domain: {peer_address}")

src/communicatorutils/restarttor.py

@@ -1,5 +1,25 @@
+"""
+Onionr - Private P2P Communication.
+
+Restart Onionr managed Tor
+"""
 import netcontroller
 import config
+"""
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+"""
+
 
 def restart(comm_inst):
     if not config.get('tor.use_existing_tor', False):

src/communicatorutils/servicecreator.py

@@ -1,9 +1,14 @@
-'''
+"""
-    Onionr - Private P2P Communication
+Onionr - Private P2P Communication.
 
-    Creates an onionr direct connection service by scanning all connection blocks
+Creates an onionr direct connection service by scanning all connection blocks
-'''
+"""
-'''
+import communicator
+from onionrblocks import onionrblockapi
+from onionrutils import stringvalidators, bytesconverter
+from coredb import blockmetadb
+from onionrservices import server_exists
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,26 +21,23 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import communicator
+
-from onionrblocks import onionrblockapi
+
-import logger
-from onionrutils import stringvalidators, bytesconverter
-from coredb import blockmetadb
-from onionrservices import server_exists
 def service_creator(daemon):
     assert isinstance(daemon, communicator.OnionrCommunicatorDaemon)
-    
+
     # Find socket connection blocks
     # TODO cache blocks and only look at recently received ones
     con_blocks = blockmetadb.get_blocks_by_type('con')
     for b in con_blocks:
-        if not b in daemon.active_services:
+        if b not in daemon.active_services:
             bl = onionrblockapi.Block(b, decrypt=True)
             bs = bytesconverter.bytes_to_str(bl.bcontent) + '.onion'
             if server_exists(bl.signer):
                 continue
-            if stringvalidators.validate_pub_key(bl.signer) and stringvalidators.validate_transport(bs):
+            if stringvalidators.validate_pub_key(bl.signer) and \
+                    stringvalidators.validate_transport(bs):
                 signer = bytesconverter.bytes_to_str(bl.signer)
                 daemon.active_services.append(b)
                 daemon.active_services.append(signer)

src/config/__init__.py

@@ -2,7 +2,11 @@
 
 This file deals with configuration management.
 """
-import os, json, logger
+import os
+from json import JSONDecodeError
+
+import ujson as json
+import logger
 import filepaths
 
 from . import onboarding
@@ -102,7 +106,7 @@ def save():
     try:
         with open(get_config_file(), 'w', encoding="utf8") as configfile:
             json.dump(get_config(), configfile, indent=2)
-    except json.JSONDecodeError:
+    except JSONDecodeError:
         logger.warn('Failed to write to configuration file.')
 
 
@@ -112,7 +116,7 @@ def reload():
     try:
         with open(get_config_file(), 'r', encoding="utf8") as configfile:
             set_config(json.loads(configfile.read()))
-    except (FileNotFoundError, json.JSONDecodeError) as e:
+    except (FileNotFoundError, JSONDecodeError) as e:
         pass
         #logger.debug('Failed to parse configuration file.')
 

src/config/onboarding.py

@@ -43,6 +43,7 @@ def set_config_from_onboarding(config_settings: OnboardingConfig):
 
     if get(config_settings, 'localThreat'):
         config.set('general.security_level', 3)
+        config.set('transports.lan', False)
 
     config.set('ui.theme', 'light')
     if get(config_settings, 'useDark'):

src/etc/onionrvalues.py

@@ -23,7 +23,7 @@ import filepaths
 DENIABLE_PEER_ADDRESS = "OVPCZLOXD6DC5JHX4EQ3PSOGAZ3T24F75HQLIUZSDSMYPEOXCPFA"
 PASSWORD_LENGTH = 25
 ONIONR_TAGLINE = 'Private P2P Communication - GPLv3 - https://Onionr.net'
-ONIONR_VERSION = '3.1.0'
+ONIONR_VERSION = '4.0.0'
 ONIONR_VERSION_CODENAME = 'Genesis'
 ONIONR_VERSION_TUPLE = tuple(ONIONR_VERSION.split('.')) # (MAJOR, MINOR, VERSION)
 API_VERSION = '1' # increments of 1; only change when something fundamental about how the API works changes. This way other nodes know how to communicate without learning too much information about you.

src/filepaths/__init__.py

@@ -36,3 +36,5 @@ onboarding_mark_file = home + 'onboarding-completed'
 log_file = home + 'onionr.log'
 
 ephemeral_services_file = home + 'ephemeral-services.list'
+
+restarting_indicator = home + "is-restarting"

src/httpapi/apiutils/getblockdata.py

@@ -1,4 +1,5 @@
-import json
+import ujson as json
+
 from onionrblocks import onionrblockapi
 from onionrutils import bytesconverter, stringvalidators
 import onionrexceptions

src/httpapi/configapi/__init__.py

@@ -17,8 +17,10 @@
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
-import json
+from json import JSONDecodeError
+import ujson as json
 from flask import Blueprint, request, Response, abort
+
 import config, onionrutils
 
 from onionrutils.bytesconverter import bytes_to_str
@@ -41,7 +43,7 @@ def set_all_config():
     """Overwrite existing JSON config with new JSON string"""
     try:
         new_config = request.get_json(force=True)
-    except json.JSONDecodeError:
+    except JSONDecodeError:
         abort(400)
     else:
         config.set_config(new_config)
@@ -58,7 +60,7 @@ def set_by_key(key):
     """
     try:
         data = json.loads(bytes_to_str(request.data))
-    except (json.JSONDecodeError, KeyError):
+    except (JSONDecodeError, KeyError):
         abort(400)
     config.set(key, data, True)
     return Response('success')

src/httpapi/friendsapi/__init__.py

@@ -1,9 +1,13 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    This file creates http endpoints for friend management
+This file creates http endpoints for friend management
-'''
+"""
-'''
+import ujson as json
+
+from onionrusers import contactmanager
+from flask import Blueprint, Response, request, abort, redirect
+from coredb import keydb
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,12 +20,7 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import json
-from onionrusers import contactmanager
-from flask import Blueprint, Response, request, abort, redirect
-from coredb import keydb
-
 friends = Blueprint('friends', __name__)
 
 @friends.route('/friends/list')

src/httpapi/insertblock.py

@@ -2,7 +2,7 @@
 
 Create blocks with the client api server
 """
-import json
+import ujson as json
 import threading
 
 from flask import Blueprint, Response, request, g

src/httpapi/miscclientapi/endpoints.py

@@ -8,7 +8,6 @@ import platform
 
 from flask import Response, Blueprint, request, send_from_directory, abort
 from flask import g
-from gevent import spawn
 import unpaddedbase32
 
 from httpapi import apiutils
@@ -20,6 +19,7 @@ from onionrutils import mnemonickeys
 from onionrutils import bytesconverter
 from etc import onionrvalues
 from utils import reconstructhash
+from utils.gettransports import get as get_tor
 """
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -40,6 +40,7 @@ pub_key = onionrcrypto.pub_key.replace('=', '')
 SCRIPT_NAME = os.path.dirname(os.path.realpath(__file__)) + \
               f'/../../../{onionrvalues.SCRIPT_NAME}'
 
+
 class PrivateEndpoints:
     def __init__(self, client_api):
         private_endpoints_bp = Blueprint('privateendpoints', __name__)
@@ -74,7 +75,6 @@ class PrivateEndpoints:
                 raise ValueError('block hash needs to be alpha numeric')
             name = reconstructhash.reconstruct_hash(name)
             if name in client_api.publicAPI.hideBlocks:
-                spawn(_delay_wait_for_share_block_removal)
                 return Response("will be removed")
             else:
                 client_api.publicAPI.hideBlocks.append(name)
@@ -140,3 +140,10 @@ class PrivateEndpoints:
         def is_tor_ready():
             """If Tor is starting up, the web UI is not ready to be used."""
             return Response(str(g.too_many.get(NetController).readyState).lower())
+
+        @private_endpoints_bp.route('/gettoraddress')
+        def get_tor_address():
+            """Return public Tor v3 Onion address for this node"""
+            if not config.get('general.security_level', 0) == 0:
+                abort(404)
+            return Response(get_tor()[0])

src/httpapi/miscpublicapi/getblocks.py

@@ -1,9 +1,16 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Public endpoints to get block data and lists
+Public endpoints to get block data and lists
-'''
+"""
-'''
+from flask import Response, abort
+
+import config
+from onionrutils import bytesconverter, stringvalidators
+from coredb import blockmetadb
+from utils import reconstructhash
+from onionrblocks import BlockList
+from .. import apiutils
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,13 +23,9 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-from flask import Response, abort
+
-import config
+
-from onionrutils import bytesconverter, stringvalidators
-from coredb import blockmetadb
-from utils import reconstructhash
-from .. import apiutils
 def get_public_block_list(publicAPI, request):
     # Provide a list of our blocks, with a date offset
     dateAdjust = request.args.get('date')
@@ -37,15 +40,16 @@ def get_public_block_list(publicAPI, request):
         share_list += '%s\n' % (reconstructhash.deconstruct_hash(b),)
     return Response(share_list)
 
+
 def get_block_data(publicAPI, data):
-    '''data is the block hash in hex'''
+    """data is the block hash in hex"""
     resp = ''
     if stringvalidators.validate_hash(data):
         if not config.get('general.hide_created_blocks', True) or data not in publicAPI.hideBlocks:
-            if data in blockmetadb.get_block_list():
+            if data in publicAPI._too_many.get(BlockList).get():
                 block = apiutils.GetBlockData().get_block_data(data, raw=True, decrypt=False)
                 try:
-                    block = block.encode() # Encode in case data is binary
+                    block = block.encode('utf-8') # Encode in case data is binary
                 except AttributeError:
                     if len(block) == 0:
                         abort(404)

src/httpapi/miscpublicapi/upload.py

@@ -1,21 +1,21 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
+
+Accept block uploads to the public API server
+"""
+import sys
 
-    Accept block uploads to the public API server
-'''
 from gevent import spawn
 from gevent import threading
-
-import sys
 from flask import Response
 from flask import abort
+from flask import g
 
 from onionrutils import localcommand
 from onionrblocks import blockimporter
 import onionrexceptions
 import logger
 
-'''
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -28,7 +28,7 @@ import logger
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
 
 
 def accept_upload(request):
@@ -40,17 +40,19 @@ def accept_upload(request):
         try:
             b_hash = blockimporter.import_block_from_data(data)
             if b_hash:
-                spawn(
+                if g.too_many.get_by_string("OnionrCommunicatorDaemon").onlinePeers:
-                    localcommand.local_command,
+                    spawn(
-                    f'/daemon-event/upload_event',
+                        localcommand.local_command,
-                    post=True,
+                        f'/daemon-event/upload_event',
-                    is_json=True,
+                        post=True,
-                    postData={'block': b_hash}
+                        is_json=True,
-                    ).get(timeout=10)
+                        post_data={'block': b_hash}
+                        ).get(timeout=10)
                 resp = 'success'
             else:
                 resp = 'failure'
-                logger.warn('Error encountered importing uploaded block')
+                logger.warn(
+                    f'Error encountered importing uploaded block {b_hash}')
         except onionrexceptions.BlacklistedBlock:
             logger.debug('uploaded block is blacklisted')
             resp = 'failure'
@@ -62,6 +64,8 @@ def accept_upload(request):
         abort(400)
     elif resp == 'proof':
         resp = Response(resp, 400)
+        logger.warn(
+            f'Error importing uploaded block, invalid proof {b_hash}')
     else:
         resp = Response(resp)
     return resp

src/httpapi/onionrsitesapi/__init__.py

@@ -1,21 +1,6 @@
-"""
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    view and interact with onionr sites
+view and interact with onionr sites
-"""
-"""
-    This program is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation, either version 3 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License
-    along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 import base64
 import binascii
@@ -30,6 +15,21 @@ import onionrexceptions
 from onionrutils import stringvalidators
 from onionrutils import mnemonickeys
 from . import sitefiles
+"""
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+"""
+
 
 site_api = Blueprint('siteapi', __name__)
 

src/httpapi/onionrsitesapi/sitefiles.py

@@ -1,21 +1,6 @@
-"""
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Read onionr site files
+Read onionr site files
-"""
-"""
-    This program is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation, either version 3 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License
-    along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 from typing import Union, Tuple
 import tarfile
@@ -32,30 +17,52 @@ from onionrblocks import insert
 from onionrtypes import UserID, DeterministicKeyPassphrase, BlockHash
 
 from onionrcrypto import generate_deterministic
+"""
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+"""
+
 
 def find_site_gzip(user_id: str)->tarfile.TarFile:
     """Return verified site tar object"""
     sites = blockmetadb.get_blocks_by_type('osite')
     user_site = None
+    unpadded_user = user_id
     user_id = unpaddedbase32.repad(user_id)
     for site in sites:
         block = onionrblockapi.Block(site)
-        if block.isSigner(user_id):
+        if block.isSigner(user_id) or block.isSigner(unpadded_user):
             user_site = block
     if not user_site is None:
         return tarfile.open(fileobj=io.BytesIO(user_site.bcontent), mode='r')
     return None
 
+
 def get_file(user_id, file)->Union[bytes, None]:
     """Get a site file content"""
     ret_data = ""
     site = find_site_gzip(user_id)
+
+    if file.endswith('/'):
+        file += 'index.html'
     if site is None: return None
     for t_file in site.getmembers():
+
         if t_file.name.replace('./', '') == file:
             return site.extractfile(t_file)
     return None
 
+
 def create_site(admin_pass: DeterministicKeyPassphrase, directory:str='.')->Tuple[UserID, BlockHash]:
     public_key, private_key = generate_deterministic(admin_pass)
 

src/httpapi/security/client.py

@@ -1,9 +1,12 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Process incoming requests to the client api server to validate they are legitimate
+Process incoming requests to the client api server to validate they are legitimate
-'''
+"""
-'''
+import hmac
+from flask import Blueprint, request, abort, g
+from onionrservices import httpheaders
+from . import pluginwhitelist
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,11 +19,7 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import hmac
-from flask import Blueprint, request, abort, g
-from onionrservices import httpheaders
-from . import pluginwhitelist
 
 # Be extremely mindful of this. These are endpoints available without a password
 whitelist_endpoints = ['www', 'staticfiles.homedata', 'staticfiles.sharedContent',
@@ -36,7 +35,7 @@ class ClientAPISecurity:
 
         @client_api_security_bp.before_app_request
         def validate_request():
-            '''Validate request has set password and is the correct hostname'''
+            """Validate request has set password and is the correct hostname"""
             # For the purpose of preventing DNS rebinding attacks
             if request.host != '%s:%s' % (client_api.host, client_api.bindPort):
                 abort(403)
@@ -66,5 +65,5 @@ class ClientAPISecurity:
             if request.endpoint in ('siteapi.site', 'siteapi.siteFile'):
                 resp.headers['Content-Security-Policy'] = "default-src 'none'; style-src 'self' data: 'unsafe-inline'; img-src 'self' data:; media-src 'self' data:"
             else:
-                resp.headers['Content-Security-Policy'] = "default-src 'none'; script-src 'self'; object-src 'none'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; media-src 'none'; frame-src 'none'; font-src 'self'; connect-src 'self'"
+                resp.headers['Content-Security-Policy'] = "default-src 'none'; script-src 'self'; object-src 'none'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; media-src 'self'; frame-src 'none'; font-src 'self'; connect-src 'self'"
             return resp

src/httpapi/security/pluginwhitelist.py

@@ -1,8 +1,8 @@
-"""
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Load web UI client endpoints into the whitelist from plugins
+Load web UI client endpoints into the whitelist from plugins
 """
+import onionrplugins
 """
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -17,7 +17,8 @@
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
-import onionrplugins
+
+
 def load_plugin_security_whitelist_endpoints(whitelist: list):
     """Accept a list reference of whitelist endpoints from security/client.py and
     append plugin's specified endpoints to them by attribute"""

src/httpapi/security/public.py

@@ -32,9 +32,12 @@ class PublicAPISecurity:
             """Validate request has the correct hostname"""
             # If high security level, deny requests to public
             # (HS should be disabled anyway for Tor, but might not be for I2P)
+
+            g.is_onionr_client = False
             transports = gettransports.get()
             if public_api.config.get('general.security_level', default=1) > 0:
                 abort(403)
+
             if request.host not in transports:
                 # Abort conn if wrong HTTP hostname, to prevent DNS rebinding
                 abort(403)
@@ -46,6 +49,11 @@ class PublicAPISecurity:
                     g.is_onionr_client = False
             except KeyError:
                 g.is_onionr_client = False
+            # Add shared objects
+            try:
+                g.too_many = public_api._too_many
+            except KeyError:
+                g.too_many = None
 
         @public_api_security_bp.after_app_request
         def send_headers(resp):
@@ -57,10 +65,12 @@ class PublicAPISecurity:
             NON_NETWORK_HEADERS = ('Content-Security-Policy', 'X-Frame-Options',
                                    'X-Content-Type-Options', 'Feature-Policy',
                                    'Clear-Site-Data', 'Referrer-Policy')
+
             try:
                 if g.is_onionr_client:
                     for header in NON_NETWORK_HEADERS: del resp.headers[header]
             except AttributeError:
                 abort(403)
+
             public_api.lastRequest = epoch.get_rounded_epoch(roundS=5)
             return resp

src/httpapi/sse/README.md

@@ -1,3 +1,6 @@
 # sse
 
-This folder contains a wrapper for handling server sent event loops
+This folder contains a wrapper for handling server sent event loops
+
+
+

src/httpapi/sse/private/__init__.py

@@ -2,10 +2,17 @@
 
 SSE API for node client access
 """
+from pathlib import Path
+
 from flask import g, Blueprint
 from gevent import sleep
 import gevent
+import ujson
 
+from onionrblocks.onionrblockapi import Block
+from coredb.dbfiles import block_meta_db
+from coredb.blockmetadb import get_block_list
+from onionrutils.epoch import get_epoch
 from onionrstatistics.transports.tor import TorStats
 from .. import wrapper
 """
@@ -45,3 +52,27 @@ def stream_tor_circuits():
             yield "data: " + tor_stats.get_json() + "\n\n"
             sleep(10)
     return SSEWrapper.handle_sse_request(circuit_stat_stream)
+
+@private_sse_blueprint.route('/recentblocks')
+def stream_recent_blocks():
+    def _compile_json(b_list):
+        js = {}
+        block_obj = None
+        for block in b_list:
+            block_obj = Block(block)
+            if block_obj.isEncrypted:
+                js[block] = 'encrypted'
+            else:
+                js[block] = Block(block).btype
+        return ujson.dumps({"blocks": js}, reject_bytes=True)
+
+    def _stream_recent():
+        last_time = Path(block_meta_db).stat().st_ctime
+        while True:
+            if Path(block_meta_db).stat().st_ctime != last_time:
+                last_time = Path(block_meta_db).stat().st_ctime
+                yield "data: " + _compile_json(get_block_list(get_epoch() - 5)) + "\n\n"
+            else:
+                yield "data: none" + "\n\n"
+            sleep(5)
+    return SSEWrapper.handle_sse_request(_stream_recent)

src/lan/__init__.py

@@ -7,7 +7,6 @@ from threading import Thread
 if TYPE_CHECKING:
     from toomanyobjs import TooMany
 
-from .client import Client
 from .discover import learn_services, advertise_service
 """
     This program is free software: you can redistribute it and/or modify
@@ -34,7 +33,6 @@ class LANManager:
 
 
     def start(self):
-        Thread(target=learn_services, args=[self.too_many.get(Client)], daemon=True).start()
+        Thread(target=learn_services, daemon=True).start()
         Thread(target=advertise_service, daemon=True).start()
-        Thread(target=self.too_many.get(Client, (self.peers,)).start, daemon=True).start()
 

src/lan/client/__init__.py

@@ -2,11 +2,18 @@
 
 LAN transport client thread
 """
-from typing import List
+import requests
 
-from onionrcrypto.cryptoutils.randomshuffle import random_shuffle
+from typing import Set
+
+from onionrtypes import LANIP
 from utils.bettersleep import better_sleep
-from onionrutils.basicrequests import do_post_request, do_get_request
+import logger
+from coredb.blockmetadb import get_block_list
+from onionrblocks.blockimporter import import_block_from_data
+from ..server import ports
+
+from threading import Thread
 """
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -22,41 +29,39 @@ from onionrutils.basicrequests import do_post_request, do_get_request
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 
-
+connected_lan_peers: Set[LANIP] = set([])
-class ConnectedError(Exception): pass
 
 
+def _lan_work(peer: LANIP):
+    def _sync_peer(url):
+        our_blocks = get_block_list()
+        blocks = requests.get(url + 'blist/0').text.splitlines()
+        for block in blocks:
+            if block not in our_blocks:
+                import_block_from_data(requests.get(url + f'get/{block}', stream=True).raw.read(6000000))
 
-class Client:
-    def __init__(self):
-        self.peers = []
-        self.lookup_time = {}
-        self.poll_delay = 10
-        self.active_threads: set = set([])
 
-    def get_lookup_time(self, peer):
+    for port in ports:
         try:
-            return self.lookup_time[peer]
+            root = f'http://{peer}:{port}/'
-        except KeyError:
+            if requests.get(f'{root}ping').text != 'onionr!':
-            return 0
+                connected_lan_peers.remove(peer)
-
+            else:
-    def peer_thread(self, peer):
+                logger.info(f'[LAN] Connected to {peer}:{port}', terminal=True)
-        def do_peer_sync(): return
+                while True:
-        if peer in self.active_threads:
+                    try:
-            raise ConnectedError
+                        _sync_peer(root)
-        self.active_threads.add(peer)
+                    except requests.exceptions.ConnectionError:
-        do_peer_sync()
+                        break
-        self.active_threads.remove(peer)
+                break
-
+        except requests.exceptions.ConnectionError:
-    def start(self):
+            pass
-        while True:
+    else:
-            peers = random_shuffle(list(set(self.peers) ^ self.active_threads))
+        connected_lan_peers.remove(peer)
-            try:
-                self.peer_thread(peers[0])
-            except IndexError:
-                pass
 
 
-
+def connect_peer(peer: LANIP):
-            better_sleep(self.poll_delay)
+    if peer not in connected_lan_peers:
+        connected_lan_peers.add(peer)
+        Thread(target=_lan_work, args=[peer], daemon=True).start()
 

src/lan/discover.py

@@ -9,8 +9,9 @@ from typing import List
 from ipaddress import ip_address
 from socket import SHUT_RDWR
 
-from .getip import lan_ips
+from .getip import lan_ips, best_ip
 from utils.bettersleep import better_sleep
+from . import client
 """
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -32,7 +33,7 @@ ANNOUNCE_LOOP_SLEEP = 30
 
 
 
-def learn_services(lan_client):
+def learn_services():
     """Take a list to infintely add lan service info to."""
 
     sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
@@ -52,17 +53,17 @@ def learn_services(lan_client):
         if 'onionr' not in service_ips:
             continue
         service_ips = service_ips.replace('onionr-', '').split('-')
+
         port = 0
         for service in service_ips:
             try:
                 ip_address(service)
                 if not ip_address(service).is_private: raise ValueError
                 if service in lan_ips: raise ValueError
-                if service in lan_client.peers: raise ValueError
             except ValueError:
-                service_ips.remove(service)
+                pass
-        p = list(lan_client.peers)
+            else:
-        lan_client.peers = list(set(service_ips + p))
+                client.connect_peer(service)
 
 
 def advertise_service(specific_ips=None):
@@ -71,10 +72,8 @@ def advertise_service(specific_ips=None):
     # for all packets sent, after three hops on the network the packet will not
     # be re-sent/broadcast (see https://www.tldp.org/HOWTO/Multicast-HOWTO-6.html)
     MULTICAST_TTL = 3
-    if specific_ips is None:
+
-        ips = '-'.join(lan_ips)
+    ips = best_ip
-    else:
-        ips = '-'.join(specific_ips)
 
     sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
     sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, MULTICAST_TTL)

src/lan/server/__init__.py

@@ -2,19 +2,26 @@
 
 LAN transport server thread
 """
+import ipaddress
+import time
+from threading import Thread
+
 from gevent.pywsgi import WSGIServer
 from flask import Flask
 from flask import Response
 from flask import request
+from flask import abort
 
 from onionrblocks.onionrblockapi import Block
 from httpapi.fdsafehandler import FDSafeHandler
 from netcontroller import get_open_port
 import config
 from coredb.blockmetadb import get_block_list
-from lan.getip import best_ip
+from lan.getip import best_ip, lan_ips
 from onionrutils import stringvalidators
 from httpapi.miscpublicapi.upload import accept_upload
+import logger
+from utils.bettersleep import better_sleep
 """
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -29,7 +36,8 @@ from httpapi.miscpublicapi.upload import accept_upload
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
-
+ports = range(1337, 1340)
+_start_time = time.time()
 
 class LANServer:
     def __init__(self, shared_state):
@@ -41,6 +49,16 @@ class LANServer:
             self.host = best_ip
         self.port = None
 
+        @app.before_request
+        def dns_rebinding_prevention():
+            if request.remote_addr in lan_ips or ipaddress.ip_address(request.remote_addr).is_loopback:
+                if time.time() - _start_time > 600:
+                    abort(403)
+            if request.host != f'{self.host}:{self.port}':
+                logger.warn('Potential DNS rebinding attack on LAN server:')
+                logger.warn(f'Hostname {request.host} was used instead of {self.host}:{self.port}')
+                abort(403)
+
         @app.route('/blist/<time>')
         def get_block_list_for_lan(time):
             return Response('\n'.join(get_block_list(dateRec=time)))
@@ -54,16 +72,32 @@ class LANServer:
 
         @app.route("/ping")
         def ping():
-            return Response("pong!")
+            return Response("onionr!")
 
         @app.route('/upload', methods=['POST'])
         def upload_endpoint():
             return accept_upload(request)
 
     def start_server(self):
-        self.server = WSGIServer((self.host, get_open_port()),
+        def _show_lan_bind(port):
-                                 self.app, log=None,
+            better_sleep(1)
-                                 handler_class=FDSafeHandler)
+            if self.server.started and port == self.server.server_port:
-        self.port = self.server.server_port
+                logger.info(f'Serving to LAN on {self.host}:{self.port}', terminal=True)
-        self.server.serve_forever()
+        if self.host == "":
-
+            logger.info("Not binding to LAN due to no private network configured.", terminal=True)
+            return
+        for i in ports:
+            self.server = WSGIServer((self.host, i),
+                                    self.app, log=None,
+                                    handler_class=FDSafeHandler)
+            self.port = self.server.server_port
+            try:
+                Thread(target=_show_lan_bind, args=[i], daemon=True).start()
+                self.server.serve_forever()
+            except OSError:
+                pass
+            else:
+                break
+        else:
+            logger.warn("Could not bind to any LAN ports " + str(min(ports)) + "-" + str(max(ports)), terminal=True)
+            return

src/logger/raw.py

@@ -1,9 +1,11 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Output raw data to file or terminal
+Output raw data to file or terminal
-'''
+"""
-'''
+import sys
+import os
+from . import settings, colors
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,14 +18,14 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import sys, os
-from . import settings, colors
 colors = colors.Colors
+
+
 def raw(data, fd = sys.stdout, terminal = False):
-    '''
+    """
         Outputs raw data to console without formatting
-    '''
+    """
 
     if terminal and (settings.get_settings() & settings.OUTPUT_TO_CONSOLE):
         try:
@@ -33,8 +35,14 @@ def raw(data, fd = sys.stdout, terminal = False):
     if settings.get_settings() & settings.OUTPUT_TO_FILE:
         fdata = ''
         try:
-            with open(settings._outputfile, 'r') as file:
+            for _ in range(5):
-                fdata = file.read()
+                try:
+                    with open(settings._outputfile, 'r') as file:
+                        fdata = file.read()
+                except UnicodeDecodeError:
+                    pass
+                else:
+                    break
         except FileNotFoundError:
             pass
         fdata = fdata + '\n' + data
@@ -43,4 +51,4 @@ def raw(data, fd = sys.stdout, terminal = False):
             fdata.pop(0)
         fdata = '\n'.join(fdata)
         with open(settings._outputfile, 'w') as file:
-            file.write(fdata)
+            file.write(fdata)

src/netcontroller/torcontrol/rebuildtor.py

@@ -30,5 +30,5 @@ def rebuild():
         f'/daemon-event/restart_tor',
         post=True,
         is_json=True,
-        postData={}
+        post_data={}
     ).get(10)

src/onionrblocks/__init__.py

@@ -1,5 +1,5 @@
 from . import insert
 from .insert import time_insert
-
+from .blocklist import BlockList
 insert = insert.insert_block
 time_insert = time_insert

src/onionrblocks/blocklist.py

@@ -0,0 +1,59 @@
+"""Onionr - Private P2P Communication.
+
+Get an auto updating list of blocks
+"""
+from threading import Thread
+
+from watchdog.observers import Observer
+from watchdog.events import FileSystemEventHandler
+
+from utils.identifyhome import identify_home
+from coredb.dbfiles import block_meta_db
+from coredb.blockmetadb import get_block_list, get_blocks_by_type
+from onionrutils.epoch import get_epoch
+"""
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+"""
+
+class BlockList:
+    def __init__(self, auto_refresh=True, block_type=''):
+        self.block_type = block_type
+        self.refresh_db()
+        self.check_time = get_epoch()
+
+        class Refresher(FileSystemEventHandler):
+            @staticmethod
+            def on_modified(event):
+                if event.src_path != block_meta_db:
+                    return
+                self.refresh_db()
+        if auto_refresh:
+            def auto_refresher():
+                observer = Observer()
+                observer.schedule(Refresher(), identify_home(), recursive=False)
+                observer.start()
+                while observer.is_alive():
+                    # call import func with timeout
+                    observer.join(120)
+            Thread(target=auto_refresher, daemon=True).start()
+
+    def get(self):
+        return self.block_list
+
+    def refresh_db(self):
+        self.check_time = get_epoch()
+        if not self.block_type:
+            self.block_list = get_block_list()
+        else:
+            self.block_list = get_blocks_by_type(self.block_type)

src/onionrblocks/insert/main.py

@@ -1,8 +1,28 @@
-"""
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Create and insert Onionr blocks
+Create and insert Onionr blocks
 """
+from typing import Union
+import ujson as json
+
+from gevent import spawn
+
+from onionrutils import bytesconverter, epoch
+import filepaths
+import onionrstorage
+from .. import storagecounter
+from onionrplugins import onionrevents as events
+from etc import powchoice, onionrvalues
+import config
+import onionrcrypto as crypto
+import onionrexceptions
+from onionrusers import onionrusers
+from onionrutils import localcommand, blockmetadata, stringvalidators
+import coredb
+from onionrproofs.vdf import multiprocess_create
+
+import logger
+from onionrtypes import UserIDSecretKey
 """
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -17,31 +37,16 @@
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
-from typing import Union
-import json
 
-from gevent import spawn
-
-from onionrutils import bytesconverter, epoch
-import filepaths, onionrstorage
-from .. import storagecounter
-from onionrplugins import onionrevents as events
-from etc import powchoice, onionrvalues
-import config, onionrcrypto as crypto, onionrexceptions
-from onionrusers import onionrusers
-from onionrutils import localcommand, blockmetadata, stringvalidators
-import coredb
-from onionrproofs.vdf import multiprocess_create
-
-import logger
-from onionrtypes import UserIDSecretKey
 
 def _check_upload_queue():
-    """Returns the current upload queue len
+    """
+    Return the current upload queue len.
+
     raises OverflowError if max, false if api not running
     """
     max_upload_queue: int = 5000
-    queue = localcommand.local_command('/gethidden', maxWait=10)
+    queue = localcommand.local_command('/gethidden', max_wait=10)
     up_queue = False
 
     try:
@@ -60,16 +65,15 @@ def insert_block(data: Union[str, bytes], header: str = 'txt',
                  expire: Union[int, None] = None, disableForward: bool = False,
                  signing_key: UserIDSecretKey = '') -> Union[str, bool]:
     """
-        Inserts a block into the network
+    Create and insert a block into the network.
-        encryptType must be specified to encrypt a block
+
-        if expire is less than date, assumes seconds into future.
+    encryptType must be specified to encrypt a block
-            if not assume exact epoch
+    if expire is less than date, assumes seconds into future.
+        if not assume exact epoch
     """
     our_private_key = crypto.priv_key
     our_pub_key = crypto.pub_key
 
-    is_offline = True
-
     storage_counter = storagecounter.StorageCounter()
 
     allocationReachedMessage = 'Cannot insert block, disk allocation reached.'
@@ -77,12 +81,11 @@ def insert_block(data: Union[str, bytes], header: str = 'txt',
         logger.error(allocationReachedMessage)
         raise onionrexceptions.DiskAllocationReached
 
-    if not _check_upload_queue() is False: is_offline = False
-
     if signing_key != '':
         # if it was specified to use an alternative private key
         our_private_key = signing_key
-        our_pub_key = bytesconverter.bytes_to_str(crypto.cryptoutils.get_pub_key_from_priv(our_private_key))
+        our_pub_key = bytesconverter.bytes_to_str(
+            crypto.cryptoutils.get_pub_key_from_priv(our_private_key))
 
     use_subprocess = powchoice.use_subprocess(config)
 
@@ -113,7 +116,8 @@ def insert_block(data: Union[str, bytes], header: str = 'txt',
     signer = ''
     metadata = {}
 
-    # metadata is full block metadata, meta is internal, user specified metadata
+    # metadata is full block metadata
+    # meta is internal, user specified metadata
 
     # only use header if not set in provided meta
 
@@ -122,9 +126,13 @@ def insert_block(data: Union[str, bytes], header: str = 'txt',
     if encryptType in ('asym', 'sym'):
         metadata['encryptType'] = encryptType
     else:
-        if not config.get('general.store_plaintext_blocks', True): raise onionrexceptions.InvalidMetadata("Plaintext blocks are disabled, yet a plaintext block was being inserted")
+        if not config.get('general.store_plaintext_blocks', True):
-        if not encryptType in ('', None):
+            raise onionrexceptions.InvalidMetadata(
-            raise onionrexceptions.InvalidMetadata('encryptType must be asym or sym, or blank')
+                "Plaintext blocks are disabled, " +
+                "yet a plaintext block was being inserted")
+        if encryptType not in ('', None):
+            raise onionrexceptions.InvalidMetadata(
+                'encryptType must be asym or sym, or blank')
 
     try:
         data = data.encode()
@@ -132,13 +140,16 @@ def insert_block(data: Union[str, bytes], header: str = 'txt',
         pass
 
     if encryptType == 'asym':
-        meta['rply'] = createTime # Duplicate the time in encrypted messages to prevent replays
+        # Duplicate the time in encrypted messages to help prevent replays
+        meta['rply'] = createTime
         if sign and asymPeer != our_pub_key:
             try:
-                forwardEncrypted = onionrusers.OnionrUser(asymPeer).forwardEncrypt(data)
+                forwardEncrypted = onionrusers.OnionrUser(
+                    asymPeer).forwardEncrypt(data)
                 data = forwardEncrypted[0]
                 meta['forwardEnc'] = True
-                expire = forwardEncrypted[2] # Expire time of key. no sense keeping block after that
+                # Expire time of key. no sense keeping block after that
+                expire = forwardEncrypted[2]
             except onionrexceptions.InvalidPubkey:
                 pass
             if not disableForward:
@@ -147,11 +158,13 @@ def insert_block(data: Union[str, bytes], header: str = 'txt',
     jsonMeta = json.dumps(meta)
     plaintextMeta = jsonMeta
     if sign:
-        signature = crypto.signing.ed_sign(jsonMeta.encode() + data, key=our_private_key, encodeResult=True)
+        signature = crypto.signing.ed_sign(
+            jsonMeta.encode() + data, key=our_private_key, encodeResult=True)
         signer = our_pub_key
 
     if len(jsonMeta) > 1000:
-        raise onionrexceptions.InvalidMetadata('meta in json encoded form must not exceed 1000 bytes')
+        raise onionrexceptions.InvalidMetadata(
+            'meta in json encoded form must not exceed 1000 bytes')
 
     # encrypt block metadata/sig/content
     if encryptType == 'sym':
@@ -160,17 +173,22 @@ def insert_block(data: Union[str, bytes], header: str = 'txt',
         if stringvalidators.validate_pub_key(asymPeer):
             # Encrypt block data with forward secrecy key first, but not meta
             jsonMeta = json.dumps(meta)
-            jsonMeta = crypto.encryption.pub_key_encrypt(jsonMeta, asymPeer, encodedData=True).decode()
+            jsonMeta = crypto.encryption.pub_key_encrypt(
-            data = crypto.encryption.pub_key_encrypt(data, asymPeer, encodedData=False)#.decode()
+                jsonMeta, asymPeer, encodedData=True).decode()
-            signature = crypto.encryption.pub_key_encrypt(signature, asymPeer, encodedData=True).decode()
+            data = crypto.encryption.pub_key_encrypt(
-            signer = crypto.encryption.pub_key_encrypt(signer, asymPeer, encodedData=True).decode()
+                data, asymPeer, encodedData=False)
+            signature = crypto.encryption.pub_key_encrypt(
+                signature, asymPeer, encodedData=True).decode()
+            signer = crypto.encryption.pub_key_encrypt(
+                signer, asymPeer, encodedData=True).decode()
             try:
                 onionrusers.OnionrUser(asymPeer, saveUser=True)
             except ValueError:
                 # if peer is already known
                 pass
         else:
-            raise onionrexceptions.InvalidPubkey(asymPeer + ' is not a valid base32 encoded ed25519 key')
+            raise onionrexceptions.InvalidPubkey(
+                asymPeer + ' is not a valid base32 encoded ed25519 key')
 
     # compile metadata
     metadata['meta'] = jsonMeta
@@ -180,16 +198,27 @@ def insert_block(data: Union[str, bytes], header: str = 'txt',
     metadata['time'] = createTime
 
     # ensure expire is integer and of sane length
-    if type(expire) is not type(None):
+    if type(expire) is not type(None):  # noqa
-        if not len(str(int(expire))) < 20: raise ValueError('expire must be valid int less than 20 digits in length')
+        if not len(str(int(expire))) < 20:
+            raise ValueError(
+                'expire must be valid int less than 20 digits in length')
         # if expire is less than date, assume seconds into future
         if expire < epoch.get_epoch():
             expire = epoch.get_epoch() + expire
         metadata['expire'] = expire
 
+<<<<<<< HEAD
     data = json.dumps(metadata).encode() + b'\n' + data
     mimc_hash = multiprocess_create(data)
     if mimc_hash != False:
+=======
+    # send block data (and metadata) to POW module to get tokenized block data
+    if use_subprocess:
+        payload = subprocesspow.SubprocessPOW(data, metadata).start()
+    else:
+        payload = onionrproofs.POW(metadata, data).waitForResult()
+    if payload != False:  # noqa
+>>>>>>> master
         try:
             retData = onionrstorage.set_data(data, mimc_hash)
         except onionrexceptions.DiskAllocationReached:
@@ -197,16 +226,21 @@ def insert_block(data: Union[str, bytes], header: str = 'txt',
             retData = False
         else:
             if disableForward:
-                logger.warn(f'{retData} asym encrypted block created w/o forward secrecy')
+                logger.warn(
-            # Tell the api server through localCommand to wait for the daemon to upload this block to make statistical analysis more difficult
+                    f'{retData} asym encrypted block created w/o ephemerality')
+            """
+            Tell the api server through localCommand to wait for the daemon to
+            upload this block to make statistical analysis more difficult
+            """
             spawn(
                 localcommand.local_command,
-                f'/daemon-event/upload_event',
+                '/daemon-event/upload_event',
                 post=True,
                 is_json=True,
-                postData={'block': retData}
+                post_data={'block': retData}
                 ).get(timeout=5)
-            coredb.blockmetadb.add.add_to_block_DB(retData, selfInsert=True, dataSaved=True)
+            coredb.blockmetadb.add.add_to_block_DB(
+                retData, selfInsert=True, dataSaved=True)
 
             if expire is None:
                 coredb.blockmetadb.update_block_info(
@@ -222,17 +256,25 @@ def insert_block(data: Union[str, bytes], header: str = 'txt',
 
             blockmetadata.process_block_metadata(retData)
 
-    if retData != False:
+    if retData != False:  # noqa
         if plaintextPeer == onionrvalues.DENIABLE_PEER_ADDRESS:
-            events.event('insertdeniable', {'content': plaintext, 'meta': plaintextMeta, 'hash': retData, 'peer': bytesconverter.bytes_to_str(asymPeer)}, threaded = True)
+            events.event('insertdeniable',
+                         {'content': plaintext, 'meta': plaintextMeta,
+                          'hash': retData,
+                          'peer': bytesconverter.bytes_to_str(asymPeer)},
+                         threaded=True)
         else:
-            events.event('insertblock', {'content': plaintext, 'meta': plaintextMeta, 'hash': retData, 'peer': bytesconverter.bytes_to_str(asymPeer)}, threaded = True)
+            events.event('insertblock',
+                         {'content': plaintext, 'meta': plaintextMeta,
+                          'hash': retData,
+                          'peer': bytesconverter.bytes_to_str(asymPeer)},
+                         threaded=True)
 
     spawn(
         localcommand.local_command,
         '/daemon-event/remove_from_insert_queue_wrapper',
         post=True,
-        postData={'block_hash': retData},
+        post_data={'block_hash': retData},
         is_json=True
         ).get(timeout=5)
     return retData

src/onionrblocks/onionrblockapi.py

@@ -1,9 +1,26 @@
-'''
+"""Onionr - P2P Anonymous Storage Network.
-    Onionr - P2P Anonymous Storage Network
 
-    This file contains the OnionrBlocks class which is a class for working with Onionr blocks
+OnionrBlocks class for abstraction of blocks
-'''
+"""
-'''
+import binascii
+import os
+import datetime
+import onionrstorage
+
+import unpaddedbase32
+import ujson as json
+import nacl.exceptions
+
+import logger
+import onionrexceptions
+from onionrusers import onionrusers
+from onionrutils import stringvalidators, epoch
+from coredb import blockmetadb
+from onionrutils import bytesconverter
+from onionrstorage import removeblock
+import onionrblocks
+from onionrcrypto import encryption, cryptoutils as cryptoutils, signing
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,18 +33,9 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import unpaddedbase32
+
-import binascii
+
-import logger, config, onionrexceptions, nacl.exceptions
-import json, os, sys, datetime, base64, onionrstorage
-from onionrusers import onionrusers
-from onionrutils import stringvalidators, epoch
-from coredb import blockmetadb
-from onionrutils import bytesconverter
-from onionrstorage import removeblock
-import onionrblocks
-from onionrcrypto import encryption, cryptoutils as cryptoutils, signing
 class Block:
     blockCacheOrder = list() # NEVER write your own code that writes to this!
     blockCache = dict() # should never be accessed directly, look at Block.getCache()
@@ -62,9 +70,9 @@ class Block:
         self.update()
 
     def decrypt(self, encodedData = True):
-        '''
+        """
             Decrypt a block, loading decrypted data into their vars
-        '''
+        """
 
         if self.decrypted:
             return True
@@ -126,9 +134,9 @@ class Block:
         return retData
 
     def verifySig(self):
-        '''
+        """
             Verify if a block's signature is signed by its claimed signer
-        '''
+        """
         if self.signer is None:
             return False
         if signing.ed_verify(data=self.signedData, key=self.signer, sig=self.signature, encodedData=True):
@@ -138,7 +146,7 @@ class Block:
         return self.validSig
 
     def update(self, data = None, file = None):
-        '''
+        """
             Loads data from a block in to the current object.
 
             Inputs:
@@ -151,7 +159,7 @@ class Block:
 
             Outputs:
             - (bool): indicates whether or not the operation was successful
-        '''
+        """
         try:
             # import from string
             blockdata = data
@@ -205,12 +213,12 @@ class Block:
         return False
 
     def delete(self):
-        '''
+        """
             Deletes the block's file and records, if they exist
 
             Outputs:
             - (bool): whether or not the operation was successful
-        '''
+        """
 
         if self.exists():
             try:
@@ -224,7 +232,7 @@ class Block:
         return False
 
     def save(self, sign = False, recreate = True):
-        '''
+        """
             Saves a block to file and imports it into Onionr
 
             Inputs:
@@ -233,7 +241,7 @@ class Block:
 
             Outputs:
             - (bool): whether or not the operation was successful
-        '''
+        """
 
         try:
             if self.isValid() is True:
@@ -253,45 +261,45 @@ class Block:
     # getters
 
     def getExpire(self):
-        '''
+        """
             Returns the expire time for a block
 
             Outputs:
             - (int): the expire time for a block, or None
-        '''
+        """
         return self.expire
 
     def getHash(self):
-        '''
+        """
             Returns the hash of the block if saved to file
 
             Outputs:
             - (str): the hash of the block, or None
-        '''
+        """
 
         return self.hash
 
     def getType(self):
-        '''
+        """
             Returns the type of the block
 
             Outputs:
             - (str): the type of the block
-        '''
+        """
         return self.btype
 
     def getRaw(self):
-        '''
+        """
             Returns the raw contents of the block, if saved to file
 
             Outputs:
             - (bytes): the raw contents of the block, or None
-        '''
+        """
 
         return self.raw
 
     def getHeader(self, key = None, default = None):
-        '''
+        """
             Returns the header information
 
             Inputs:
@@ -299,7 +307,7 @@ class Block:
 
             Outputs:
             - (dict/str): either the whole header as a dict, or one value
-        '''
+        """
 
         if not key is None:
             if key in self.getHeader():
@@ -308,7 +316,7 @@ class Block:
         return self.bheader
 
     def getMetadata(self, key = None, default = None):
-        '''
+        """
             Returns the metadata information
 
             Inputs:
@@ -316,7 +324,7 @@ class Block:
 
             Outputs:
             - (dict/str): either the whole metadata as a dict, or one value
-        '''
+        """
 
         if not key is None:
             if key in self.getMetadata():
@@ -325,77 +333,77 @@ class Block:
         return self.bmetadata
 
     def getContent(self):
-        '''
+        """
             Returns the contents of the block
 
             Outputs:
             - (str): the contents of the block
-        '''
+        """
 
         return self.bcontent
 
     def getDate(self):
-        '''
+        """
             Returns the date that the block was received, if loaded from file
 
             Outputs:
             - (datetime): the date that the block was received
-        '''
+        """
 
         return self.date
 
     def getBlockFile(self):
-        '''
+        """
             Returns the location of the block file if it is saved
 
             Outputs:
             - (str): the location of the block file, or None
-        '''
+        """
 
         return self.blockFile
 
     def isValid(self):
-        '''
+        """
             Checks if the block is valid
 
             Outputs:
             - (bool): whether or not the block is valid
-        '''
+        """
 
         return self.valid
 
     def isSigned(self):
-        '''
+        """
             Checks if the block was signed
 
             Outputs:
             - (bool): whether or not the block is signed
-        '''
+        """
 
         return self.signed
 
     def getSignature(self):
-        '''
+        """
             Returns the base64-encoded signature
 
             Outputs:
             - (str): the signature, or None
-        '''
+        """
 
         return self.signature
 
     def getSignedData(self):
-        '''
+        """
             Returns the data that was signed
 
             Outputs:
             - (str): the data that was signed, or None
-        '''
+        """
 
         return self.signedData
 
     def isSigner(self, signer, encodedData = True):
-        '''
+        """
             Checks if the block was signed by the signer inputted
 
             Inputs:
@@ -404,7 +412,7 @@ class Block:
 
             Outputs:
             - (bool): whether or not the signer of the block is the signer inputted
-        '''
+        """
         signer = unpaddedbase32.repad(bytesconverter.str_to_bytes(signer))
         try:
             if (not self.isSigned()) or (not stringvalidators.validate_pub_key(signer)):
@@ -417,7 +425,7 @@ class Block:
     # setters
 
     def setType(self, btype):
-        '''
+        """
             Sets the type of the block
 
             Inputs:
@@ -425,13 +433,13 @@ class Block:
 
             Outputs:
             - (Block): the Block instance
-        '''
+        """
 
         self.btype = btype
         return self
 
     def setMetadata(self, key, val):
-        '''
+        """
             Sets a custom metadata value
 
             Metadata should not store block-specific data structures.
@@ -442,13 +450,13 @@ class Block:
 
             Outputs:
             - (Block): the Block instance
-        '''
+        """
 
         self.bmetadata[key] = val
         return self
 
     def setContent(self, bcontent):
-        '''
+        """
             Sets the contents of the block
 
             Inputs:
@@ -456,14 +464,14 @@ class Block:
 
             Outputs:
             - (Block): the Block instance
-        '''
+        """
 
         self.bcontent = str(bcontent)
         return self
 
     # static functions
     def exists(bHash):
-        '''
+        """
             Checks if a block is saved to file or not
 
             Inputs:
@@ -473,7 +481,7 @@ class Block:
 
             Outputs:
             - (bool): whether or not the block file exists
-        '''
+        """
 
         # no input data? scrap it.
         if bHash is None:

src/onionrcommands/daemonlaunch/__init__.py

@@ -25,16 +25,15 @@ from utils import identifyhome
 import filepaths
 from etc import onionrvalues, cleanup
 from onionrcrypto import getourkeypair
-from utils import hastor, logoheader
+from utils import hastor
 import runtests
 from httpapi import daemoneventsapi
 from .. import version
 from .getapihost import get_api_host_until_available
 from utils.bettersleep import better_sleep
 from netcontroller.torcontrol.onionservicecreator import create_onion_service
-from .quotes import QUOTE
 from .killdaemon import kill_daemon  # noqa
-from utils.boxprint import bordered
+from .showlogo import show_logo
 from lan import LANManager
 from lan.server import LANServer
 from sneakernet import sneakernet_import_thread
@@ -60,8 +59,60 @@ def _proper_shutdown():
     sys.exit(1)
 
 
+def _show_info_messages():
+    version.version(verbosity=5, function=logger.info)
+    logger.debug('Python version %s' % platform.python_version())
+
+    if onionrvalues.DEVELOPMENT_MODE:
+        logger.warn('Development mode enabled', timestamp=False, terminal=True)
+
+    logger.info('Using public key: %s' %
+                (logger.colors.underline +
+                 getourkeypair.get_keypair()[0][:52]))
+
+def _setup_online_mode(use_existing_tor: bool, 
+                   net: NetController,
+                   security_level: int):
+    if config.get('transports.tor', True):
+        # If we are using tor, check if we are using an existing tor instance
+        # if we are, we need to create an onion service on it and set attrs on our NetController
+        # if not, we need to tell netcontroller to start one
+        if use_existing_tor:
+            try:
+                os.mkdir(filepaths.tor_hs_loc)
+            except FileExistsError:
+                pass
+            net.socksPort = config.get('tor.existing_socks_port')
+            try:
+                net.myID = create_onion_service(
+                    port=net.apiServerIP + ':' + str(net.hsPort))[0]
+            except IncorrectPassword:
+                # Exit if we cannot connect to the existing Tor instance
+                logger.error('Invalid Tor control password', terminal=True)
+                localcommand.local_command('shutdown')
+                cleanup.delete_run_files()
+                sys.exit(1)
+
+            if not net.myID.endswith('.onion'):
+                net.myID += '.onion'
+            with open(filepaths.tor_hs_address_file, 'w') as tor_file:
+                tor_file.write(net.myID)
+        else:
+            logger.info('Tor is starting...', terminal=True)
+            if not net.startTor():
+                # Exit if we cannot start Tor.
+                localcommand.local_command('shutdown')
+                cleanup.delete_run_files()
+                sys.exit(1)
+        if len(net.myID) > 0 and security_level == 0:
+            logger.debug('Started .onion service: %s' %
+                        (logger.colors.underline + net.myID))
+        else:
+            logger.debug('.onion service disabled')
+
+
 def daemon():
-    """Start the Onionr communication daemon."""
+    """Start Onionr's primary threads for communicator, API server, node, and LAN."""
     # Determine if Onionr is in offline mode.
     # When offline, Onionr can only use LAN and disk transport
     offline_mode = config.get('general.offline_mode', False)
@@ -90,33 +141,22 @@ def daemon():
 
     # Init run time tester
     # (ensures Onionr is running right, for testing purposes)
-
+    # Run time tests are not normally run
     shared_state.get(runtests.OnionrRunTestManager)
+
+    # Create singleton 
     shared_state.get(serializeddata.SerializedData)
 
     shared_state.share_object()  # share the parent object to the threads
 
+    show_logo()
+
+    # since we randomize loopback API server hostname to protect against attacks,
+    # we have to wait for it to become set
     apiHost = ''
     if not offline_mode:
         apiHost = get_api_host_until_available()
 
-    logger.raw('', terminal=True)
-    # print nice header thing :)
-    if config.get('general.display_header', True):
-        logoheader.header("")
-        if QUOTE[1]:
-            logger.info(
-                "\u001b[33m\033[F" + bordered(QUOTE[0] + '\n -' + QUOTE[1]),
-                terminal=True)
-        else:
-            logger.info("\u001b[33m\033[F" + bordered(QUOTE[0]), terminal=True)
-
-    version.version(verbosity=5, function=logger.info)
-    logger.debug('Python version %s' % platform.python_version())
-
-    if onionrvalues.DEVELOPMENT_MODE:
-        logger.warn('Development mode enabled', timestamp=False, terminal=True)
-
     net = NetController(config.get('client.public.port', 59497),
                         apiServerIP=apiHost)
     shared_state.add(net)
@@ -127,43 +167,11 @@ def daemon():
     use_existing_tor = config.get('tor.use_existing_tor', False)
 
     if not offline_mode:
-        if config.get('transports.tor', True):
+        # we need to setup tor for use
-            if use_existing_tor:
+        _setup_online_mode(use_existing_tor, net, security_level)
-                try:
+    
-                    os.mkdir(filepaths.tor_hs_loc)
+    
-                except FileExistsError:
+    _show_info_messages()
-                    pass
-                net.socksPort = config.get('tor.existing_socks_port')
-                try:
-                    net.myID = create_onion_service(
-                        port=net.apiServerIP + ':' + str(net.hsPort))[0]
-                except IncorrectPassword:
-                    logger.error('Invalid Tor control password', terminal=True)
-                    localcommand.local_command('shutdown')
-                    cleanup.delete_run_files()
-                    sys.exit(1)
-
-                if not net.myID.endswith('.onion'):
-                    net.myID += '.onion'
-                with open(filepaths.tor_hs_address_file, 'w') as tor_file:
-                    tor_file.write(net.myID)
-            else:
-                logger.info('Tor is starting...', terminal=True)
-                if not net.startTor():
-                    localcommand.local_command('shutdown')
-                    cleanup.delete_run_files()
-                    sys.exit(1)
-            if len(net.myID) > 0 and security_level == 0:
-                logger.debug('Started .onion service: %s' %
-                            (logger.colors.underline + net.myID))
-            else:
-                logger.debug('.onion service disabled')
-
-    logger.info('Using public key: %s' %
-                (logger.colors.underline +
-                 getourkeypair.get_keypair()[0][:52]))
-
-    better_sleep(1)
 
     events.event('init', threaded=False)
     events.event('daemon_start')
@@ -175,6 +183,7 @@ def daemon():
         Thread(target=sneakernet_import_thread, daemon=True).start()
 
     Thread(target=statistics_reporter, args=[shared_state], daemon=True).start()
+
     communicator.startCommunicator(shared_state)
 
     clean_ephemeral_services()
@@ -190,7 +199,7 @@ def daemon():
     better_sleep(5)
 
     cleanup.delete_run_files()
-    if config.get('general.security_level', 1) >= 2:
+    if security_level >= 2:
         filenuke.nuke.clean_tree(identifyhome.identify_home())
 
 
@@ -205,6 +214,13 @@ def start(override: bool = False):
     Error exit if there is and its not overridden
     """
     if os.path.exists(filepaths.lock_file) and not override:
+        if os.path.exists(filepaths.restarting_indicator):
+            try:
+                os.remove(filepaths.restarting_indicator)
+            except FileNotFoundError:
+                pass
+            else:
+                return
         logger.fatal('Cannot start. Daemon is already running,'
                      + ' or it did not exit cleanly.\n'
                      + ' (if you are sure that there is not a daemon running,'

src/onionrcommands/daemonlaunch/quotes.py

@@ -6,8 +6,8 @@ QUOTES = [
     "Katherine Neville"),
     ("Hack the Planet",
     ""),
-    ("Live long and prosper",
+    ("Study after study has show that human behavior changes when we know we’re being watched.\nUnder observation, we act less free, which means we effectively *are* less free.",
-    "Spock"),
+    "Edward Snowdwen"),
     ("A revolution without dancing is a revolution not worth having",
     "V for Vendetta"),
     ("There can be no justice so long as laws are absolute. Even life itself is an exercise in exceptions",

src/onionrcommands/daemonlaunch/showlogo.py

@@ -0,0 +1,46 @@
+"""Onionr - Private P2P Communication.
+
+Show nice logo
+"""
+import os
+
+import config
+import logger
+
+from .quotes import QUOTE
+from utils.boxprint import bordered
+from utils import logoheader
+"""
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+"""
+
+
+def show_logo():
+    logger.raw('', terminal=True)
+    # print nice header thing :)
+    if config.get('general.display_header', True):
+        logoheader.header("")
+        if os.get_terminal_size().columns >= 120:
+            if QUOTE[1]:  # If there is an author to show for the quote
+                logger.info(
+                    "\u001b[33m\033[F" + bordered(QUOTE[0] + '\n -' + QUOTE[1]),
+                    terminal=True)
+            else:
+                logger.info("\u001b[33m\033[F" + bordered(QUOTE[0]), terminal=True)
+        else:
+            if QUOTE[1]:
+                logger.info("\u001b[33m\033[F" + QUOTE[0] + '\n -' + QUOTE[1],
+                            terminal=True)
+            else:
+                logger.info("\u001b[33m\033[F" + QUOTE[0], terminal=True)

src/onionrcommands/restartonionr.py

@@ -49,9 +49,10 @@ def restart():
             return
     except (AttributeError, OSError):
         logger.warn('Could not fork on restart')
-
+    with open(filepaths.restarting_indicator, 'w') as f:
+        f.write('t')
     daemonlaunch.kill_daemon()
-    while localcommand.local_command('ping', maxWait=8) == 'pong!':
+    while localcommand.local_command('ping', max_wait=8) == 'pong!':
         time.sleep(0.3)
     time.sleep(15)
     while (os.path.exists(filepaths.private_API_host_file) or

src/onionrcommands/runtimetestcmd.py

@@ -28,8 +28,8 @@ def do_runtime_test():
         f'daemon-event/test_runtime',
         post=True,
         is_json=True,
-        postData={},
+        post_data={},
-        maxWait=300
+        max_wait=300
     ).get(300)
 
 

src/onionrcommands/sitecreator.py

@@ -3,6 +3,7 @@
 Command to create Onionr mutli-page sites
 """
 import sys
+import os
 import getpass
 
 from httpapi import onionrsitesapi
@@ -27,8 +28,11 @@ from etc import onionrvalues
 def create_multipage_site():
     """Command to create mutlipage sites with specified dir and password."""
     error_encountered = False
+    orig_dir = os.getcwd()
     try:
         directory = sys.argv[2]
+        os.chdir(directory)
+        directory = '.'
     except IndexError:
         directory = '.'
     try:
@@ -51,7 +55,7 @@ If you want to update your site later you must remember the passphrase.''',
         error_encountered = True
         logger.error(
             f'Passphrase must be at least {onionrvalues.PASSWORD_LENGTH}' +
-            'characters.', terminal=True)
+            ' characters.', terminal=True)
 
     if error_encountered:
         sys.exit(1)
@@ -61,6 +65,7 @@ If you want to update your site later you must remember the passphrase.''',
     results = (results[0].replace('=', ''), results[1])
     logger.info(f'Site address {results[0]}', terminal=True)
     logger.info(f'Block for this version {results[1]}', terminal=True)
+    os.chdir(orig_dir)
 
 
 create_multipage_site.onionr_help = "[directory path "  # type: ignore

src/onionrcrypto/cryptoutils/verifypow.py

@@ -31,7 +31,7 @@ def verify_POW(blockContent):
         # logger.debug('Validated block pow')
         retData = True
     else:
-        logger.debug("Invalid token, bad proof")
+        logger.debug(f"Invalid token, bad proof for {blockHash} {puzzle}")
         raise onionrexceptions.InvalidProof('Proof for %s needs to be %s' % (blockHash, puzzle))
 
     return retData

src/onionrproofs/__init__.py

@@ -1,9 +1,18 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Proof of work module
+Proof of work module
-'''
+"""
-'''
+import multiprocessing, time, math, threading, binascii, sys, json
+import nacl.encoding, nacl.hash, nacl.utils
+
+import config
+import logger
+from onionrblocks import onionrblockapi, storagecounter
+from onionrutils import bytesconverter
+from onionrcrypto import hashers
+
+from .blocknoncestart import BLOCK_NONCE_START_INT
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,31 +25,22 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import multiprocessing, time, math, threading, binascii, sys, json
-import nacl.encoding, nacl.hash, nacl.utils
-
-import config, logger
-from onionrblocks import onionrblockapi, storagecounter
-from onionrutils import bytesconverter
-from onionrcrypto import hashers
-
-from .blocknoncestart import BLOCK_NONCE_START_INT
 config.reload()
 
 def getDifficultyModifier():
-    '''returns the difficulty modifier for block storage based
+    """returns the difficulty modifier for block storage based
     on a variety of factors, currently only disk use.
-    '''
+    """
     percentUse = storagecounter.StorageCounter().get_percent()
     difficultyIncrease = math.floor(4 * percentUse) # difficulty increase is a step function
 
     return difficultyIncrease
 
 def getDifficultyForNewBlock(data):
-    '''
+    """
     Get difficulty for block. Accepts size in integer, Block instance, or str/bytes full block contents
-    '''
+    """
     if isinstance(data, onionrblockapi.Block):
         dataSizeInBytes = len(bytesconverter.str_to_bytes(data.getRaw()))
     else:
@@ -54,15 +54,15 @@ def getDifficultyForNewBlock(data):
     return retData
 
 def getHashDifficulty(h: str):
-    '''
+    """
         Return the amount of leading zeroes in a hex hash string (hexHash)
-    '''
+    """
     return len(h) - len(h.lstrip('0'))
 
 def hashMeetsDifficulty(hexHash):
-    '''
+    """
         Return bool for a hash string to see if it meets pow difficulty defined in config
-    '''
+    """
     hashDifficulty = getHashDifficulty(hexHash)
 
     try:
@@ -72,97 +72,3 @@ def hashMeetsDifficulty(hexHash):
 
     return hashDifficulty >= expected
 
-
-class POW:
-    def __init__(self, metadata, data, threadCount = 1, minDifficulty=0):
-        self.foundHash = False
-        self.difficulty = 0
-        self.data = data
-        self.metadata = metadata
-        self.threadCount = threadCount
-        self.hashing = False
-
-        json_metadata = json.dumps(metadata).encode()
-
-        try:
-            self.data = self.data.encode()
-        except AttributeError:
-            pass
-
-        if minDifficulty > 0:
-            self.difficulty = minDifficulty
-        else:
-            # Calculate difficulty. Dumb for now, may use good algorithm in the future.
-            self.difficulty = getDifficultyForNewBlock(bytes(json_metadata + b'\n' + self.data))
-
-        logger.info('Computing POW (difficulty: %s)...' % (self.difficulty,))
-
-        self.mainHash = '0' * 64
-        self.puzzle = self.mainHash[0:min(self.difficulty, len(self.mainHash))]
-        for i in range(max(1, threadCount)):
-            t = threading.Thread(name = 'thread%s' % i, target = self.pow, args = (True,))
-            t.start()
-
-    def pow(self, reporting = False):
-        startTime = math.floor(time.time())
-        self.hashing = True
-        self.reporting = reporting
-        iFound = False # if current thread is the one that found the answer
-        nonce = BLOCK_NONCE_START_INT
-        while self.hashing:
-            self.metadata['pow'] = nonce
-            payload = json.dumps(self.metadata).encode() + b'\n' + self.data
-            token = hashers.sha3_hash(payload)
-            try:
-                # on some versions, token is bytes
-                token = token.decode()
-            except AttributeError:
-                pass
-            if self.puzzle == token[0:self.difficulty]:
-                self.hashing = False
-                iFound = True
-                self.result = payload
-                break
-            nonce += 1
-
-        if iFound:
-            endTime = math.floor(time.time())
-            if self.reporting:
-                logger.debug('Found token after %s seconds: %s' % (endTime - startTime, token), timestamp=True)
-
-    def shutdown(self):
-        self.hashing = False
-        self.puzzle = ''
-
-    def changeDifficulty(self, newDiff):
-        self.difficulty = newDiff
-
-    def getResult(self):
-        '''
-            Returns the result then sets to false, useful to automatically clear the result
-        '''
-
-        try:
-            retVal = self.result
-        except AttributeError:
-            retVal = False
-
-        self.result = False
-        return retVal
-
-    def waitForResult(self):
-        '''
-            Returns the result only when it has been found, False if not running and not found
-        '''
-        result = False
-        try:
-            while True:
-                result = self.getResult()
-                if not self.hashing:
-                    break
-                else:
-                    time.sleep(1)
-        except KeyboardInterrupt:
-            self.shutdown()
-            logger.warn('Got keyboard interrupt while waiting for POW result, stopping')
-        return result

src/onionrproofs/subprocesspow.py

@@ -1,22 +1,20 @@
 #!/usr/bin/env python3
-"""
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Multiprocess proof of work
+Multiprocess proof of work
 """
 
 import os
 from multiprocessing import Pipe, Process
 import threading
 import time
-import json
+import onionrproofs
-import secrets
+
+import ujson as json
 
 import logger
-import onionrproofs
 import onionrcrypto as crypto
 from onionrutils import bytesconverter
-from .blocknoncestart import BLOCK_NONCE_START_INT
 
 """
     This program is free software: you can redistribute it and/or modify
@@ -110,11 +108,12 @@ class SubprocessPOW:
 
     def do_pow(self, pipe):
         """find partial hash colision generating nonce for a block"""
-        nonce = -secrets.randbelow(10**10)
+        nonce = 0
         data = self.data
         metadata = self.metadata
         puzzle = self.puzzle
         difficulty = self.difficulty
+
         while True:
             # Break if shutdown received
             if pipe.poll() and pipe.recv() == 'shutdown':

src/onionrsetup/setupconfig.py

@@ -1,9 +1,19 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Initialize Onionr configuration
+Initialize Onionr configuration
-'''
+"""
-'''
+import os
+import base64
+
+import ujson as json
+
+import config
+import logger
+import netcontroller
+from etc import onionrvalues
+from logger.settings import *
+from utils import readstatic
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,15 +26,10 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import os, json, base64
+
-import config, logger, netcontroller
-from etc import onionrvalues
-from logger.settings import *
-from utils import readstatic
 
 def setup_config():
-    
     if not os.path.exists(config._configfile):
         # this is the default config, it will be overwritten if a config file already exists. Else, it saves it
         conf_data = readstatic.read_static('default_config.json', ret_bin=False)

src/onionrstatistics/devreporting/__init__.py

@@ -1,10 +1,29 @@
+"""Onionr - Private P2P Communication.
+
+Dev utility to profile an Onionr subnetwork.
+"""
+import ujson as json
+
 import config
 from utils.bettersleep import better_sleep
 from utils.gettransports import get as get_transports
 from onionrutils import basicrequests
 from onionrutils import epoch
+"""
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+"""
 
-import json
 
 def statistics_reporter(shared_state):
     server = config.get('statistics.server', '')

src/onionrstatistics/serializeddata.py

@@ -2,10 +2,10 @@
 
 Serialize various node information
 """
-import json
 from gevent import sleep
 
 from psutil import Process, WINDOWS
+import ujson as json
 
 from coredb import blockmetadb
 from utils.sizeutils import size, human_size

src/onionrstatistics/transports/tor/__init__.py

@@ -2,8 +2,7 @@
 
 
 """
-import json
+import ujson as json
-
 from stem import CircStatus
 
 from netcontroller.torcontrol.torcontroller import get_controller

src/onionrtypes/__init__.py

@@ -3,6 +3,8 @@ from typing import NewType
 UserID = NewType('UserID', str)
 UserIDSecretKey = NewType('UserIDSecretKey', str)
 
+LANIP = NewType('LANIP', 'str')
+
 DeterministicKeyPassphrase = NewType('DeterministicKeyPassphrase', str)
 
 BlockHash = NewType('BlockHash', str)

src/onionrusers/contactmanager.py

@@ -1,9 +1,19 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Sets more abstract information related to a peer. Can be thought of as traditional 'contact' system
+Set more abstract information related to a peer.
-'''
+Can be thought of as traditional 'contact' system
-'''
+"""
+import os
+
+import ujson as json
+import unpaddedbase32
+
+import onionrexceptions
+from onionrusers import onionrusers
+from onionrutils import bytesconverter, epoch
+from utils import identifyhome
+from onionrutils import mnemonickeys
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,16 +26,9 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import os, json
+
-import unpaddedbase32
-import niceware
 
-import onionrexceptions
-from onionrusers import onionrusers
-from onionrutils import bytesconverter, epoch
-from utils import identifyhome
-from onionrutils import mnemonickeys
 class ContactManager(onionrusers.OnionrUser):
     def __init__(self, publicKey, saveUser=False, recordExpireSeconds=5):
         try:
@@ -43,10 +46,10 @@ class ContactManager(onionrusers.OnionrUser):
         self.recordExpire = recordExpireSeconds
         self.data = self._loadData()
         self.deleted = False
-        
+
         if not os.path.exists(self.dataDir):
             os.mkdir(self.dataDir)
-    
+
     def _writeData(self):
         data = json.dumps(self.data)
         with open(self.dataFile, 'w') as dataFile:
@@ -68,7 +71,7 @@ class ContactManager(onionrusers.OnionrUser):
         if autoWrite:
             self._writeData()
         return
-    
+
     def get_info(self, key, forceReload=False):
         if self.deleted:
             raise onionrexceptions.ContactDeleted

src/onionrusers/onionrusers.py

@@ -1,9 +1,20 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    Contains abstractions for interacting with users of Onionr
+Contains abstractions for interacting with users of Onionr
-'''
+"""
-'''
+import sqlite3
+import time
+
+import onionrexceptions
+from onionrutils import stringvalidators, bytesconverter, epoch
+
+import unpaddedbase32
+import nacl.exceptions
+
+from coredb import keydb, dbfiles
+import onionrcrypto
+from onionrcrypto import getourkeypair
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,14 +27,8 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import logger, onionrexceptions, json, sqlite3, time
+
-from onionrutils import stringvalidators, bytesconverter, epoch
-import unpaddedbase32
-import nacl.exceptions
-from coredb import keydb, dbfiles
-import onionrcrypto
-from onionrcrypto import getourkeypair
 
 def deleteExpiredKeys():
     # Fetch the keys we generated for the peer, that are still around
@@ -37,6 +42,7 @@ def deleteExpiredKeys():
     conn.close()
     return
 
+
 def deleteTheirExpiredKeys(pubkey):
     conn = sqlite3.connect(dbfiles.user_id_info_db, timeout=10)
     c = conn.cursor()
@@ -49,17 +55,19 @@ def deleteTheirExpiredKeys(pubkey):
     conn.commit()
     conn.close()
 
+
 DEFAULT_KEY_EXPIRE = 604800
 
+
 class OnionrUser:
 
     def __init__(self, publicKey, saveUser=False):
-        '''
+        """
             OnionrUser is an abstraction for "users" of the network.
 
             Takes a base32 encoded ed25519 public key, and a bool saveUser
             saveUser determines if we should add a user to our peer database or not.
-        '''
+        """
         publicKey = unpaddedbase32.repad(bytesconverter.str_to_bytes(publicKey)).decode()
 
         self.trust = 0
@@ -75,11 +83,11 @@ class OnionrUser:
         return
 
     def setTrust(self, newTrust):
-        '''Set the peers trust. 0 = not trusted, 1 = friend, 2 = ultimate'''
+        """Set the peers trust. 0 = not trusted, 1 = friend, 2 = ultimate"""
         keydb.userinfo.set_user_info(self.publicKey, 'trust', newTrust)
 
     def isFriend(self):
-        if keydb.userinfo.set_peer_info(self.publicKey, 'trust') == 1:
+        if keydb.userinfo.get_user_info(self.publicKey, 'trust') == 1:
             return True
         return False
 
@@ -227,4 +235,4 @@ class OnionrUser:
         friendList = []
         for x in keydb.listkeys.list_peers(trust=1):
             friendList.append(cls(x))
-        return list(friendList)
+        return list(friendList)

src/onionrutils/basicrequests.py

@@ -57,7 +57,7 @@ def do_post_request(url, data={}, port=0, proxyType='tor', max_size=10000, conte
     return retData
 
 
-def do_get_request(url, port=0, proxyType='tor', ignoreAPI=False, returnHeaders=False, max_size=5242880):
+def do_get_request(url, port=0, proxyType='tor', ignoreAPI=False, returnHeaders=False, max_size=5242880, connect_timeout=15):
     '''
     Do a get request through a local tor or i2p instance
     '''
@@ -72,7 +72,7 @@ def do_get_request(url, port=0, proxyType='tor', ignoreAPI=False, returnHeaders=
     elif proxyType == 'lan':
         address = urlparse(url).hostname
         if IPv4Address(address).is_private and not IPv4Address(address).is_loopback:
-            proxies = {}
+            proxies = None
         else:
             return
     else:
@@ -80,8 +80,9 @@ def do_get_request(url, port=0, proxyType='tor', ignoreAPI=False, returnHeaders=
     headers = {'User-Agent': 'PyOnionr', 'Connection':'close'}
     response_headers = dict()
     try:
-        proxies = {'http': 'socks4a://127.0.0.1:' + str(port), 'https': 'socks4a://127.0.0.1:' + str(port)}
+        if not proxies is None:
-        r = streamedrequests.get(url, request_headers=headers, allow_redirects=False, proxy=proxies, connect_timeout=15, stream_timeout=120, max_size=max_size)
+            proxies = {'http': 'socks4a://127.0.0.1:' + str(port), 'https': 'socks4a://127.0.0.1:' + str(port)}
+        r = streamedrequests.get(url, request_headers=headers, allow_redirects=False, proxy=proxies, connect_timeout=connect_timeout, stream_timeout=120, max_size=max_size)
         # Check server is using same API version as us
         if not ignoreAPI:
             try:

src/onionrutils/blockmetadata/fromdata.py

@@ -1,9 +1,12 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
     Return a useful tuple of (metadata (header), meta, and data) by accepting raw block data
-'''
+"""
-'''
+from json import JSONDecodeError
+import ujson as json
+
+from onionrutils import bytesconverter
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,17 +19,15 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
 
-import json
 
-from onionrutils import bytesconverter
 def get_block_metadata_from_data(block_data):
-    '''
+    """
-        accepts block contents as string, returns a tuple of 
+        accepts block contents as string, returns a tuple of
-        metadata, meta (meta being internal metadata, which will be 
+        metadata, meta (meta being internal metadata, which will be
         returned as an encrypted base64 string if it is encrypted, dict if not).
-    '''
+    """
     meta = {}
     metadata = {}
     data = block_data
@@ -37,7 +38,7 @@ def get_block_metadata_from_data(block_data):
 
     try:
         metadata = json.loads(bytesconverter.bytes_to_str(block_data[:block_data.find(b'\n')]))
-    except json.decoder.JSONDecodeError:
+    except JSONDecodeError:
         pass
     else:
         data = block_data[block_data.find(b'\n'):]

src/onionrutils/escapeansi.py

@@ -1,4 +1,6 @@
 import re
+
+
 def escape_ANSI(line):
     '''
         Remove ANSI escape codes from a string with regex

src/onionrutils/localcommand.py

@@ -1,16 +1,16 @@
-"""
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    send a command to the local API server
+send a command to the local API server
 """
-import urllib, time
+import urllib
-import json
+import time
-import functools
-from typing import TYPE_CHECKING, Callable
 
 import requests
+import deadsimplekv
+
+import logger
+import config
 
-import logger, config, deadsimplekv
 from . import getclientapiserver
 import filepaths
 """
@@ -29,11 +29,14 @@ import filepaths
 """
 config.reload()
 
-cache = deadsimplekv.DeadSimpleKV(filepaths.cached_storage, refresh_seconds=1000)
+cache = deadsimplekv.DeadSimpleKV(filepaths.cached_storage,
+                                  refresh_seconds=1000)
+
+
 def get_hostname():
     hostname = ''
     waited = 0
-    maxWait = 3
+    max_wait = 3
     while True:
         if cache.get('client_api') is None:
             try:
@@ -47,21 +50,22 @@ def get_hostname():
             hostname = cache.get('hostname')
         if hostname == '' or hostname is None:
             time.sleep(1)
-            if waited == maxWait:
+            if waited == max_wait:
                 return False
         else:
             return hostname
 
-def local_command(command, data='', silent = True, post=False,
+
-                  postData = {}, maxWait=20,
+def local_command(command, data='', silent=True, post=False,
+                  post_data={}, max_wait=20,
                   is_json=False
                   ):
-    """
+    """Send a command to the local http API server, securely.
-        Send a command to the local http API server, securely. Intended for local clients, DO NOT USE for remote peers.
+    Intended for local clients, DO NOT USE for remote peers."""
-    """
-    # TODO: URL encode parameters, just as an extra measure. May not be needed, but should be added regardless.
     hostname = get_hostname()
-    if hostname == False: return False
+    # if the api host cannot be reached, return False
+    if not hostname:
+        return False
 
     if data != '':
         data = '&data=' + urllib.parse.quote_plus(data)
@@ -74,21 +78,26 @@ def local_command(command, data='', silent = True, post=False,
             if is_json:
                 ret_data = requests.post(
                     payload,
-                    json=postData,
+                    json=post_data,
                     headers={'token': config.get('client.webpassword'),
                              'Connection': 'close'},
-                    timeout=(maxWait, maxWait)).text
+                    timeout=(max_wait, max_wait)).text
             else:
                 ret_data = requests.post(
                     payload,
-                    data=postData,
+                    data=post_data,
                     headers={'token': config.get('client.webpassword'),
                              'Connection': 'close'},
-                    timeout=(maxWait, maxWait)).text
+                    timeout=(max_wait, max_wait)).text
         else:
-            ret_data = requests.get(payload, headers={'token': config.get('client.webpassword'), 'Connection':'close'}, timeout=(maxWait, maxWait)).text
+            ret_data = requests.get(payload,
+                                    headers={'token':
+                                             config.get('client.webpassword'),
+                                             'Connection': 'close'},
+                                    timeout=(max_wait, max_wait)).text
     except Exception as error:
         if not silent:
-            logger.error('Failed to make local request (command: %s):%s' % (command, error), terminal=True)
+            logger.error('Failed to make local request (command: %s):%s' %
+                         (command, error), terminal=True)
         ret_data = False
     return ret_data

src/onionrutils/validatemetadata.py

@@ -1,8 +1,14 @@
-"""
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    validate new block's metadata
+validate new block's metadata
 """
+from json import JSONDecodeError
+import ujson as json
+
+import logger, onionrexceptions
+from etc import onionrvalues
+from . import stringvalidators, epoch, bytesconverter
+import config, filepaths, onionrcrypto
 """
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -17,11 +23,7 @@
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
-import json
+
-import logger, onionrexceptions
-from etc import onionrvalues
-from . import stringvalidators, epoch, bytesconverter
-import config, filepaths, onionrcrypto
 
 def validate_metadata(metadata, block_data) -> bool:
     """Validate metadata meets onionr spec (does not validate proof value computation), take in either dictionary or json string"""
@@ -33,7 +35,7 @@ def validate_metadata(metadata, block_data) -> bool:
     if type(metadata) is str:
         try:
             metadata = json.loads(metadata)
-        except json.JSONDecodeError:
+        except JSONDecodeError:
             pass
 
     # Validate metadata dict for invalid keys to sizes that are too large

src/runtests/__init__.py

@@ -16,6 +16,7 @@ from .clearnettor import test_clearnet_tor_request
 from .housekeeping import test_inserted_housekeeping
 from .lanservertest import test_lan_server
 from .sneakernettest import test_sneakernet_import
+from .dnsrebindingtest import test_dns_rebinding
 """
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
@@ -41,7 +42,8 @@ RUN_TESTS = [uicheck.check_ui,
              test_clearnet_tor_request,
              test_inserted_housekeeping,
              test_lan_server,
-             sneakernettest.test_sneakernet_import
+             sneakernettest.test_sneakernet_import,
+             test_dns_rebinding
              ]
 
 SUCCESS_FILE = os.path.dirname(os.path.realpath(__file__)) + '/../../tests/runtime-result.txt'

src/runtests/dnsrebindingtest.py

@@ -0,0 +1,46 @@
+"""Onionr - Private P2P Communication.
+
+Test apis for dns rebinding
+"""
+import config
+import requests
+from filepaths import private_API_host_file, public_API_host_file
+import logger
+"""
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+"""
+
+
+def test_dns_rebinding(test_manager):
+    f = ''
+    with open(private_API_host_file, 'r') as f:
+        host = f.read()
+    private_api_port = config.get('client.client.port')
+
+    if requests.get(f'http://{host}:{private_api_port}/ping', headers={'host': 'example.com'}) == 'pong!':
+        raise ValueError('DNS rebinding failed')
+    logger.info('It is normal to see 403 errors right now', terminal=True)
+
+    if config.get('general.security_level', 0) > 0 or not config.get('transports.tor', True):
+        return
+    public_api_port = config.get('client.public.port')
+    f = ''
+    with open(public_API_host_file, 'r') as f:
+        host = f.read()
+
+    if requests.get(f'http://{host}:{public_api_port}/ping', headers={'host': 'example.com'}) == 'pong!':
+        raise ValueError('DNS rebinding failed')
+    logger.info('It is normal to see 403 errors right now', terminal=True)
+
+

src/runtests/inserttest.py

@@ -12,7 +12,10 @@ def _check_remote_node(testmanager):
 def insert_bin_test(testmanager):
     data = os.urandom(32)
     b_hash = onionrblocks.insert(data)
+    time.sleep(0.3)
+    if b_hash not in testmanager._too_many.get_by_string("PublicAPI").hideBlocks:
+        raise ValueError("Block not hidden")
 
-    if not b_hash in coredb.blockmetadb.get_block_list():
+    if b_hash not in coredb.blockmetadb.get_block_list():
         logger.error(str(b_hash) + 'is not in bl')
         raise ValueError

src/runtests/lanservertest.py

@@ -10,9 +10,15 @@ import logger
 
 def test_lan_server(testmanager):
     start_time = get_epoch()
-    for i in range(1024, 65536):
+    for i in range(1337, 1340):
         try:
-            if requests.get(f"http://{best_ip}:{i}/ping").text == 'pong!':
+            if not best_ip or not best_ip.startswith(('192.168')):
+                logger.warn(
+                    "lanservertest not running, not in standard 192.168 lan " +
+                    "run this test on a lan before release",
+                    terminal=True)
+                return
+            if requests.get(f"http://{best_ip}:{i}/ping").text == 'onionr!':
                 bl = insert('test data')
                 sleep(10)
                 bl2 = insert('test data2')
@@ -20,15 +26,16 @@ def test_lan_server(testmanager):
                 bl3 = insert('test data3')
                 l = requests.get(f"http://{best_ip}:{i}/blist/0").text.split('\n')
                 if bl not in l or bl2 not in l or bl3 not in l:
-                    logger.error('blocks not in blist ' + '-'.join(l), terminal=True)
+                    logger.error('blocks not in blist ' + '-'.join(l))
                     raise ValueError
                 time = blockmetadb.get_block_date(bl3) - 1
                 l = requests.get(f"http://{best_ip}:{i}/blist/{time}").text.split('\n')
 
                 if (bl in l and bl2 in l and bl3 in l) or len(l) == 0:
-                    logger.error('Failed to get appopriate time' + '-'.join(l), terminal=True)
+                    logger.error('Failed to get appopriate time' + '-'.join(l))
                     raise ValueError
-                if onionrblockapi.Block(bl).raw != requests.get(f"http://{best_ip}:{i}/get/{bl}").content:
+                if onionrblockapi.Block(bl).raw != requests.get(f"http://{best_ip}:{i}/get/{bl}", stream=True).raw.read(6000000):
+                    logger.error('Block doesn\'t match')
                     raise ValueError
 
                 break

src/utils/colors.py

@@ -0,0 +1,60 @@
+'''
+    Onionr - Private P2P Communication
+
+    class to access ANSI control codes
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import re
+class Colors:
+    '''
+        This class allows you to set the color if ANSI codes are supported
+    '''
+    reset='\033[0m'
+    bold='\033[01m'
+    disable='\033[02m'
+    underline='\033[04m'
+    reverse='\033[07m'
+    strikethrough='\033[09m'
+    invisible='\033[08m'
+    italics='\033[3m'
+    class fg:
+        black='\033[30m'
+        red='\033[31m'
+        green='\033[32m'
+        orange='\033[33m'
+        blue='\033[34m'
+        purple='\033[35m'
+        cyan='\033[36m'
+        lightgrey='\033[37m'
+        darkgrey='\033[90m'
+        lightred='\033[91m'
+        lightgreen='\033[92m'
+        yellow='\033[93m'
+        lightblue='\033[94m'
+        pink='\033[95m'
+        lightcyan='\033[96m'
+    class bg:
+        black='\033[40m'
+        red='\033[41m'
+        green='\033[42m'
+        orange='\033[43m'
+        blue='\033[44m'
+        purple='\033[45m'
+        cyan='\033[46m'
+        lightgrey='\033[47m'
+    @staticmethod
+    def filter(data):
+        return re.compile(r'\x1B\[[0-?]*[ -/]*[@-~]').sub('', str(data))

src/utils/gettransports.py

@@ -21,11 +21,28 @@ import filepaths
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 
-files = [filepaths.tor_hs_address_file]
+files = []
 
 
+class _GetTor:
+    def __init__(self):
+        self.tor_hs = None
+
+    def get(self):
+        if self.tor_hs is None:
+            try:
+                with open(filepaths.tor_hs_address_file, 'r') as transport_file:
+                    self.tor_hs = transport_file.read().strip()
+                if not self.tor_hs:
+                    self.tor_hs = None
+            except FileNotFoundError:
+                pass
+        return self.tor_hs
+
+_tor_getter = _GetTor()
+
 def get():
-    transports = []
+    transports = [_tor_getter.get()]
     for file in files:
         try:
             with open(file, 'r') as transport_file:

start-many-nodes.py

@@ -0,0 +1,8 @@
+import os
+n = int(input("how many nodes: "))
+
+
+
+for _ in range(n):
+    os.system('./start-ram.sh')
+

static-data/bootstrap-nodes.txt

@@ -1 +1,5 @@
-3msj7fgyxgpfsjvvtcji7a4tkjbna6jmpealv6mun7435jjyptctfxyd.onion,chz7aarrmhxnefa6jx7ai3h3f5oy4sz5x4o5bbhfcq4xr3zbvsynaoad.onion,llqcrrf5cdk7p277eynepnvoo4ggrnybmp2daqtsr2hshitlmvbipdqd.onion,lqyhqt5mtsvu5bdatn4ntaplsfsrfxzp3j6gze77g4nptpxe36q7poad.onion
+<<<<<<< HEAD
+3msj7fgyxgpfsjvvtcji7a4tkjbna6jmpealv6mun7435jjyptctfxyd.onion,chz7aarrmhxnefa6jx7ai3h3f5oy4sz5x4o5bbhfcq4xr3zbvsynaoad.onion,llqcrrf5cdk7p277eynepnvoo4ggrnybmp2daqtsr2hshitlmvbipdqd.onion,lqyhqt5mtsvu5bdatn4ntaplsfsrfxzp3j6gze77g4nptpxe36q7poad.onion
+=======
+csb2thc5yzv2gbhoozbqrzv747irs5z2lbpd7eiyh6eivvltok76qrqd.onion,chz7aarrmhxnefa6jx7ai3h3f5oy4sz5x4o5bbhfcq4xr3zbvsynaoad.onion,ueawiiskhaxdhkqjvgz6drrlf7srvaifrewnb6rxf6tro3welajvlgyd.onion,lqyhqt5mtsvu5bdatn4ntaplsfsrfxzp3j6gze77g4nptpxe36q7poad.onion
+>>>>>>> master

static-data/default-plugins/chat/controlapi.py

@@ -1,9 +1,14 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    HTTP endpoints for controlling IMs
+HTTP endpoints for controlling IMs
-'''
+"""
-'''
+import ujson as json
+
+from flask import Response, request, redirect, Blueprint, send_from_directory
+import deadsimplekv as simplekv
+
+import filepaths
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,11 +21,7 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import json
-from flask import Response, request, redirect, Blueprint, send_from_directory
-import deadsimplekv as simplekv
-import filepaths
 flask_blueprint = Blueprint('chat_control', __name__)
 key_store = simplekv.DeadSimpleKV(filepaths.cached_storage, refresh_seconds=5)
 @flask_blueprint.route('/chatapi/ping')

static-data/default-plugins/chat/main.py

@@ -19,7 +19,7 @@
 '''
 
 # Imports some useful libraries
-import locale, sys, os, threading, json
+import locale, sys, os, threading, ujson as json
 locale.setlocale(locale.LC_ALL, '')
 import onionrservices, logger, config
 from onionrservices import bootstrapservice

static-data/default-plugins/chat/peerserver.py

@@ -1,9 +1,19 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    HTTP endpoints for communicating with peers
+HTTP endpoints for communicating with peers
-'''
+"""
-'''
+import sys
+import os
+from json import JSONDecodeError
+
+import deadsimplekv as simplekv
+import ujson as json
+from flask import Response, request, redirect, Blueprint, abort, g
+
+from utils import identifyhome
+from onionrutils import localcommand
+import filepaths
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,12 +26,7 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import sys, os, json
-from utils import identifyhome
-from onionrutils import localcommand
-import deadsimplekv as simplekv, filepaths
-from flask import Response, request, redirect, Blueprint, abort, g
 sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
 direct_blueprint = Blueprint('chat', __name__)
 
@@ -46,14 +51,15 @@ def sendto():
     """Endpoint peers send chat messages to"""
     try:
         msg = request.get_json(force=True)
-    except json.JSONDecodeError:
+    except JSONDecodeError:
         msg = ''
     else:
         msg = json.dumps(msg)
-        localcommand.local_command('/chat/addrec/%s' % (g.peer,), post=True, postData=msg)
+        localcommand.local_command('/chat/addrec/%s' % (g.peer,), post=True, post_data=msg)
     return Response('success')
 
 @direct_blueprint.route('/chat/poll')
 def poll_chat():
     """Endpoints peers get new messages from"""
-    return Response(localcommand.local_command('/chat/gets/%s' % (g.peer,)))
+    return Response(localcommand.local_command('/chat/gets/%s' % (g.peer,)))
+    

static-data/default-plugins/circles/flowapi.py

@@ -3,9 +3,10 @@
 This file primarily serves to allow specific fetching of circles board messages
 """
 import operator
-import json
 import os
 
+import ujson as json
+
 from flask import Response, Blueprint
 from deadsimplekv import DeadSimpleKV
 
@@ -31,7 +32,7 @@ with open(
     os.path.dirname(
         os.path.realpath(__file__)) + '/info.json', 'r') as info_file:
     data = info_file.read().strip()
-    version = json.loads(data, strict=False)['version']
+    version = json.loads(data)['version']
 
 BOARD_CACHE_FILE = identifyhome.identify_home() + '/board-index.cache.json'
 

static-data/default-plugins/circles/main.py

@@ -106,7 +106,7 @@ class OnionrFlow:
                     content = block.getContent()
                     # Escape new lines, remove trailing whitespace, and escape ansi sequences
                     content = escapeansi.escape_ANSI(content.replace(
-                        '\n', '\\n').replace('\r', '\\r').strip())
+                        b'\n', b'\\n').replace(b'\r', b'\\r').strip().decode('utf-8'))
                     logger.info(block.getDate().strftime(
                         "%m/%d %H:%M") + ' - ' + logger.colors.reset + content, prompt=False, terminal=True)
                     self.alreadyOutputed.append(b_hash)

static-data/default-plugins/encrypt/main.py

@@ -19,12 +19,20 @@
 '''
 
 # Imports some useful libraries
-import logger, config, threading, time, datetime, sys, json
+import logger, config, threading, time, datetime, sys
+
+import ujson as json
+from nacl.exceptions import TypeError as NaclTypeError
+
 from onionrutils import stringvalidators, bytesconverter
 from onionrcrypto import encryption, keypair, signing, getourkeypair
 import onionrexceptions, onionrusers
+
 import locale
 locale.setlocale(locale.LC_ALL, '')
+
+import binascii
+
 plugin_name = 'encrypt'
 
 class PlainEncryption:
@@ -107,7 +115,12 @@ class PlainEncryption:
         return
 
 def on_decrypt_cmd(api, data=None):
-    PlainEncryption(api).decrypt()
+    try:
+        PlainEncryption(api).decrypt()
+    except binascii.Error:
+        logger.error("Invalid ciphertext padding", terminal=True)
+    except NaclTypeError:
+        logger.error("Ciphertext too short.", terminal=True)
 
 def on_encrypt_cmd(api, data=None):
     PlainEncryption(api).encrypt()

static-data/default-plugins/pms/mailapi.py

@@ -1,9 +1,22 @@
-'''
+"""Onionr - Private P2P Communication.
-    Onionr - Private P2P Communication
 
-    HTTP endpoints for mail plugin.
+HTTP endpoints for mail plugin
-'''
+"""
-'''
+import sys
+import os
+
+import ujson as json
+from flask import Response, request, redirect, Blueprint, abort
+import deadsimplekv as simplekv
+
+from onionrusers import contactmanager
+from onionrutils import stringvalidators
+from utils import reconstructhash, identifyhome
+
+sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
+import loadinbox
+import sentboxdb
+"""
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
@@ -16,17 +29,7 @@
 
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-import sys, os, json
-from flask import Response, request, redirect, Blueprint, abort
-from onionrusers import contactmanager
-from onionrutils import stringvalidators
-from utils import reconstructhash, identifyhome
-import filepaths
-import deadsimplekv as simplekv
-sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
-import loadinbox, sentboxdb
-
 flask_blueprint = Blueprint('mail', __name__)
 kv = simplekv.DeadSimpleKV(identifyhome.identify_home() + '/mailcache.dat')
 

static-data/default-plugins/pms/main.py

@@ -5,7 +5,8 @@ Private messages in an email like fashion
 import locale
 import sys
 import os
-import json
+
+import ujson as json
 
 from onionrusers import contactmanager
 from utils import reconstructhash
@@ -61,6 +62,7 @@ def on_insertblock(api, data={}):
 def on_processblocks(api, data=None):
     if data['type'] != 'pm':
         return
+    notification_func = notifier.notify
     data['block'].decrypt()
     metadata = data['block'].bmetadata
 
@@ -72,7 +74,14 @@ def on_processblocks(api, data=None):
     else:
         signer = signer[:5]
 
+
     if data['block'].decrypted:
         config.reload()
+
+        if config.get('mail.notificationSound', True):
+            notification_func = notifier.notification_with_sound
         if config.get('mail.notificationSetting', True):
-            notifier.notification_with_sound(title="Onionr Mail - New Message", message="From: %s\n\nSubject: %s" % (signer, metadata['subject']))
+            if not config.get('mail.strangersNotification', True):
+                if not user.isFriend():
+                    return
+            notification_func(title="Onionr Mail - New Message", message="From: %s\n\nSubject: %s" % (signer, metadata['subject']))

static-data/default_config.json

@@ -71,6 +71,7 @@
         "tor": true
     },
     "ui": {
+        "animated_background": true,
         "theme": "dark"
     }
 }

static-data/www/board/autorefresh.js

@@ -34,4 +34,11 @@ function setupInterval(){
 var refreshInterval = setInterval(autoRefresh, 3000)
 setupInterval()
 
-checkbox.onchange = function(){setupInterval}
+checkbox.onchange = function(){setupInterval}
+
+
+document.addEventListener("visibilitychange", function() {
+    if (document.visibilityState === 'visible') {
+      autoRefresh()
+    }
+  })

static-data/www/mail/index.html

@@ -113,25 +113,6 @@
             </div>
         </div>
     </div>
-    <div id="sendMessage" class="overlay">
-        <div class="overlayContent">
-            <div class="field">
-                <label><i class="fas fa-user"></i> Select friend: <select id="friendSelect"></select></label>
-            </div>
-            <form method="post" action="" id="sendForm" enctype="application/x-www-form-urlencoded">
-                <span class="closeOverlay" overlay="sendMessage"></span>
-                <div class="field">
-                    To: <input id="draftID" type="text" name="to" placeholder="pubkey or select above" required>
-                </div>
-                Subject: <input name="subject" id="draftSubject" maxlength="25" type="text"
-                    placeholder="message subject">
-                <div class="field">
-                    <textarea name="message" class="textarea" placeholder="type your message..." id="draftText" required></textarea>
-                </div>
-                <input type="submit" value="Send" class="button is-success successBtn">
-            </form>
-        </div>
-    </div>
 
     <div id="settingsModal" class="modal">
         <div class="modal-background"></div>
@@ -180,13 +161,13 @@
             </div>
             <div class="columns notificationSetting">
                 <div class="column">
-                    Only show notifications for friends
+                    Notifications for stranger's messages
                 </div>
                 <div class="column is-2">
                     <div class="field">
-                        <input id="friendsOnlyNotifications" type="checkbox"
+                        <input id="strangersNotification" type="checkbox"
-                        class="switch is-rounded">
+                        class="switch is-rounded" checked>
-                        <label for="friendsOnlyNotifications"></label>
+                        <label for="strangersNotification"></label>
                     </div>
                 </div>
             </div>
@@ -196,9 +177,9 @@
                 </div>
                 <div class="column is-2">
                   <div class="field">
-                    <input id="audibleNotificationSetting" type="checkbox"
+                    <input id="notificationSound" type="checkbox"
-                    class="switch is-rounded">
+                    class="switch is-rounded" checked>
-                    <label for="audibleNotificationSetting"></label>
+                    <label for="notificationSound"></label>
                   </div>
                 </div>
               </div>
@@ -255,7 +236,24 @@
             </div>
         </div>
     </div>
-
+    <div id="sendMessage">
+        <div class="container">
+            <div class="field">
+                <label><i class="fas fa-user"></i> Select friend: <select id="friendSelect"></select></label>
+            </div>
+            <form method="post" action="" id="sendForm" enctype="application/x-www-form-urlencoded">
+                <div class="field">
+                    To: <input id="draftID" type="text" name="to" placeholder="pubkey or select above" required>
+                </div>
+                Subject: <input name="subject" id="draftSubject" maxlength="25" type="text"
+                    placeholder="message subject">
+                <div class="field">
+                    <textarea name="message" class="textarea" placeholder="type your message..." id="draftText" required></textarea>
+                </div>
+                <input type="submit" value="Send" class="button is-primary successBtn">
+            </form>
+        </div>
+    </div>
     <div id="infoOverlay" class="overlay">
     </div>
 </body>

static-data/www/mail/loadsettings.js

@@ -17,7 +17,13 @@ fetch('/config/get/mail', {
     if (mailSettings.notificationSetting === false){
       document.getElementById('notificationSetting').checked = false
     }
+    if (mailSettings.notificationSound === false){
+      document.getElementById('notificationSound').checked = false
+    }
     if (typeof mailSettings.signature != undefined && mailSettings.signature != null && mailSettings.signature != ""){
       document.getElementById('mailSignatureSetting').value = mailSettings.signature
     }
+    if (mailSettings.strangersNotification == false){
+      document.getElementById('strangersNotification').checked = false
+    }
   })

static-data/www/mail/mail.css

@@ -58,4 +58,3 @@
 #settingsModal small{
     font-size: 0.5em;
 }
-

static-data/www/mail/mail.js

@@ -128,6 +128,7 @@ function openThread(bHash, sender, date, sigBool, pubkey, subjectLine){
         sigEl.innerText = sigMsg
         overlay('messageDisplay')
         replyBtn.onclick = function(){
+            document.getElementById('messageDisplay').style.visibility = 'hidden'
             openReply(bHash, messageDisplay.innerText, subjectLine)
         }
         addUnknownContact.onclick = function(){
@@ -144,6 +145,7 @@ function setActiveTab(tabName){
     threadPart.innerHTML = ""
     noInbox.style.display = 'none'
     window.inboxActive = false
+    document.getElementById('sendMessage').classList.add('is-hidden')
     switch(tabName){
         case 'inbox':
             window.inboxActive = true
@@ -154,8 +156,7 @@ function setActiveTab(tabName){
             getSentbox()
             break
         case 'compose':
-            overlay('sendMessage')
+            document.getElementById('sendMessage').classList.remove('is-hidden')
-            document.getElementById('inboxTab').click()
             break
         case 'settings':
             document.getElementById('settingsModal').classList.add('is-active')
@@ -419,7 +420,6 @@ fetch('/friends/list', {
 .then(function(resp) {
     var friendSelectParent = document.getElementById('friendSelect')
     var keys = [];
-    var friend
     for(var k in resp) keys.push(k);
 
     friendSelectParent.appendChild(document.createElement('option'))
@@ -441,4 +441,10 @@ setActiveTab('inbox')
 setInterval(function(){mailPing()}, 10000)
 mailPing()
 window.inboxInterval = setInterval(function(){refreshPms(true)}, 3000)
-refreshPms(true)
+refreshPms(true)
+
+document.addEventListener("visibilitychange", function() {
+    if (document.visibilityState === 'visible') {
+        refreshPms()
+    }
+  })

static-data/www/mail/settings.js

@@ -17,6 +17,8 @@
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */
 var notificationSetting = document.getElementById('notificationSetting')
+var friendOnlyNotification = document.getElementById('strangersNotification')
+var notificationSound = document.getElementById('notificationSound')
 var sigSetting = document.getElementById('mailSignatureSetting')
 
 document.getElementById('forwardSecrecySetting').onchange = function(e){
@@ -37,6 +39,40 @@ document.getElementById('forwardSecrecySetting').onchange = function(e){
       })
 }
 
+notificationSound.onchange = function(e){
+    var postData = JSON.stringify({"notificationSound": e.target.checked})
+    fetch('/config/set/mail', {
+        method: 'POST',
+        body: postData,
+        headers: {
+          "content-type": "application/json",
+          "token": webpass
+        }})
+    .then(function(data) {
+        mailSettings['notificationSound'] = notificationSound.checked
+        PNotify.success({
+            text: 'Successfully notification sound'
+        })
+      })
+}
+
+friendOnlyNotification.onchange = function(e){
+    var postData = JSON.stringify({"strangersNotification": e.target.checked})
+    fetch('/config/set/mail', {
+        method: 'POST',
+        body: postData,
+        headers: {
+          "content-type": "application/json",
+          "token": webpass
+        }})
+    .then(function(data) {
+        mailSettings['strangersNotification'] = friendOnlyNotification.checked
+        PNotify.success({
+            text: 'Successfully toggled notifications from strangers'
+        })
+      })
+}
+
 
 notificationSetting.onchange = function(e){
     var notificationSettings = document.getElementsByClassName('notificationSetting')

static-data/www/private/index.html

@@ -15,10 +15,12 @@
     <script defer src="/shared/node_modules/pnotify/dist/iife/PNotify.js"></script>
     <script defer src="/shared/node_modules/pnotify/dist/iife/PNotifyButtons.js"></script>
     <script defer src="/shared/eventsource.js"></script>
+    <script defer src="/shared/main/particles.js"></script>
     <script defer src="/shared/loadabout.js"></script>
     <script defer src="/shared/misc.js"></script>
     <script defer src="/shared/getos.js"></script>
     <script defer src="/shared/main/stats.js"></script>
+    <script defer src="/shared/main/recent.js"></script>
     <script defer src="/shared/main/torstats.js"></script>
     <script defer src="/shared/panel.js"></script>
     <script defer src="/shared/configeditor.js"></script>
@@ -28,6 +30,7 @@
     <script defer src="/private/js/motd.js"></script>
     <script defer src="/shared/navbar.js"></script>
     <script defer src="/private/js/windows.js"></script>
+    <script defer src="/shared/main/loadTransport.js"></script>
     <script>alert("Content security policy appears to not be working. Your browser security is weak!")</script>
 </head>
 
@@ -79,25 +82,6 @@
                         </h2>
                     </div>
                     <div class="column is-7">
-                        <div class="field">
-                            <div class="field has-addons">
-                                <p class="control">
-                                    <a class="button is-static">
-                                        <i class="fas fa-fingerprint"></i>
-                                    </a>
-                                </p>
-                                <p class="control is-expanded">
-                                    <input id="myPub" class="input myPub" type="text" readonly>
-                                </p>
-                                <p class="control">
-                                    <a id="myPubCopy" class="button is-primary">
-                                        <span class="icon">
-                                            <i class="fas fa-copy"></i>
-                                        </span>
-                                    </a>
-                                </p>
-                            </div>
-                        </div>
                         <div class="field is-grouped is-grouped-centered">
                             <p class="control">
                                 <a class="button is-danger is-outlined" id="shutdownNode">
@@ -118,8 +102,9 @@
 
     <br>
 
+
     <!--Start of content-->
-    <div class="container">
+    <div class="container mainCont">
         <div class="columns">
             <div class="column">
                 <!--Onionr Card-->
@@ -199,6 +184,26 @@
                     </header>
                     <div class="card-content">
                         <div class="content">
+                            <div class="columns">
+                                <div class="column">
+                                    <div class="field">
+                                        <div class="field has-addons">
+                                            <p class="control">
+                                                <a class="button is-static">
+                                                    <i class="fas fa-fingerprint"></i>
+                                                </a>
+                                            </p>
+                                            <p class="control is-expanded">
+                                                <input id="myPub" class="input myPub" type="text" readonly>
+                                            </p>
+                                            <p class="control">
+                                                <a id="myPubCopy" class="button is-primary"><i class="fas fa-copy"></i></a>
+                                            </p>
+                                        </div>
+                                    </div>
+                                </div>
+ 
+                            </div>
                             <div class="columns">
                                 <div class="column">
                                     <span class="icon">
@@ -245,6 +250,21 @@
                                      Total Requests: <span id="totalRec">None since start</span>
                                 </div>
                             </div>
+
+                            <div class="field has-addons torTransportField">
+                                <p class="control">
+                                    <a class="button is-static">
+                                        <i class="fas fa-adjust"></i>
+                                    </a>
+                                </p>
+                                <p class="control is-expanded">
+                                    <input class="input myTor" type="text" readonly>
+                                </p>
+                                <p class="control">
+                                    <a class="button is-primary myTorCopy"><i class="fas fa-copy"></i></a>
+                                </p>
+                            </div>
+
                             <i class="fas fa-link"></i>
                             Outgoing Connections:
                             <div class="control">
@@ -254,19 +274,31 @@
                             <h4>Blocks</h4>
                             <div class="columns">
                                 <div class="column">
-                                    <i class="fas fa-database"></i>
+                                    <i class="fas fa-hdd"></i>
-                                    Stored Blocks: <span id="storedBlocks"></span>
+                                    Stored blocks: <span id="storedBlocks"></span>
                                 </div>
                                 <div class="column">
                                     <i class="fas fa-mail-bulk"></i>
-                                    Blocks in queue: <span id="blockQueue"></span>
+                                    Blocks to download: <span id="blockQueue"></span>
+                                </div>
+                            </div>
+                            <div class="columns">
+                                <div class="column">
+                                    <p class="buttons">
+                                        <button class="button is-small recentBlocksBtn">
+                                          <span class="icon is-small">
+                                            <i class="fas fa-stream"></i>
+                                          </span>
+                                          <span>Recent blocks</span>
+                                        </button>
+                                    </p>
                                 </div>
                             </div>
                             <h4>Process</h4>
                             <div class="columns">
                                 <div class="column">
                                     <i class="fas fa-microchip"></i>
-                                    Current CPU threads: <span id="threads"></span>
+                                    Current threads: <span id="threads"></span>
                                 </div>
                                 <div class="column">
                                     <i class="fas fa-memory"></i>
@@ -300,6 +332,8 @@
         </div>
     </div>
     </div>
+    </div>
+    <div id="particles-js"></div>
     <br>
     <div class="modal aboutModal">
         <div class="modal-background"></div>
@@ -311,7 +345,19 @@
             Loading... <i class="fas fa-spinner fa-spin"></i>
           </section>
         </div>
-      </div>
+    </div>
+    <div class="modal recentModal">
+        <div class="modal-background"></div>
+        <div class="modal-card">
+          <header class="modal-card-head">
+            <button class="closeRecentModal delete" aria-label="close"></button>
+          </header>
+          <section class="modal-card-body recentBody">
+                Keep this open to see new blocks as they come in (or are created)!
+              <pre class="recentBlockList"></pre>
+          </section>
+        </div>
+    </div>
 </body>
 
 </html>

static-data/www/private/main.css

@@ -32,4 +32,32 @@
 
 .torStats{
     color: red;
-}
+}
+
+/* ---- reset ---- */
+
+
+  canvas {
+    display: block;
+    vertical-align: bottom;
+  }
+  
+  /* ---- particles.js container ---- */
+  
+  #particles-js {
+    position:absolute;
+    width:100%;
+    height:100%;
+    top:0;
+    z-index:-1;
+  }
+  
+  .mainCont{
+      z-index: 2;
+      position: fixed;
+      
+  }
+
+  .torTransportField {
+      margin-top: 2em;
+  }