Merge branch 'master' of ssh://gitlab.com/beardog/onionr

.github/FUNDING.yml

@@ -0,0 +1,4 @@
+# These are supported funding model platforms
+
+ko_fi: beardogkf
+

.gitignore

@@ -18,6 +18,9 @@ core
 venv/*
 onionr/fs*
 
+*.dll
+*.exe
+
 # log files
 output.log
 *.log

CODE_OF_CONDUCT.md

@@ -40,7 +40,7 @@ Project maintainers who do not follow or enforce the Code of Conduct in good fai
 
 ## Attribution
 
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://contributor-covenant.org/version/1/4][version]
 
-[homepage]: http://contributor-covenant.org
+[homepage]: https://contributor-covenant.org
-[version]: http://contributor-covenant.org/version/1/4/
+[version]: https://contributor-covenant.org/version/1/4/

CONTRIBUTING.md

@@ -1,12 +1,12 @@
 # Contributing to Onionr
 
-One of the great things about open source projects is that they allow for many people to contribute to the project.
+One of the great things about open source projects is that they allow for anyone to contribute to the project.
 
 This file serves to provide guidelines on how to successfully contribute to Onionr.
 
 ## Code of Conduct
 
-See our [Code of Conduct](https://github.com/beardog108/onionr/blob/master/CODE_OF_CONDUCT.md)
+Contributors in project-related spaces/contexts are expected to follow the [Code of Conduct](https://github.com/beardog108/onionr/blob/master/CODE_OF_CONDUCT.md)
 
 ## Reporting Bugs
 
@@ -23,11 +23,13 @@ Please provide the following information when reporting a bug:
 
 If a bug is a security issue, please contact us privately.
 
-And most importantly, please be patient. Onionr is an open source project done by volunteers.
+And most importantly, please be patient. Onionr is a free open source project maintained by volunteers in their free time.
 
 ## Asking Questions
 
-If you need help with Onionr, you can contact the devs (be polite and remember this is a volunteer-driven non-profit project).
+If you need help with Onionr, you can contact the devs (be polite and remember this is a volunteer-driven project).
+
+Do your best to use good english.
 
 ## Contributing Code
 

LICENSE.txt

@@ -1,6 +1,3 @@
-The Onionr logo was created by [Anhar Ismail](https://github.com/anharismail) under the [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/). 
-
-
                     GNU GENERAL PUBLIC LICENSE
                        Version 3, 29 June 2007
 

README.md

@@ -5,28 +5,30 @@
 </p>
 
 <p align="center">
-    Anonymous P2P storage network 🕵️
+    Private P2P Communication Network 🕵️
 </p>
 
 (***pre-alpha & experimental, not well tested or easy to use yet***)
 
 [![Open Source Love](https://badges.frapsoft.com/os/v3/open-source.png?v=103)](https://github.com/ellerbrock/open-source-badges/)
-<img src='https://gitlab.com/beardog/Onionr/badges/master/build.svg'>
+<img src='https://gitlab.com/beardog/Onionr/badges/master/build.svg'> - [Onionr.net](https://onionr.net/) - [.onion](http://onionr.onionkvc5ibm37bmxwr56bdxcdnb6w3wm4bdghh5qo6f6za7gn7styid.onion/)
 
 <hr>
 
 **The main repository for this software is at https://gitlab.com/beardog/Onionr/**
 
-# Summary
+# About
 
-Onionr is a decentralized, peer-to-peer data storage network, designed to be anonymous and resistant to (meta)data analysis and spam/disruption.
+Onionr is a decentralized, peer-to-peer communication network, designed to be anonymous and resistant to (meta)data analysis, spam, and corruption.
 
-Onionr stores data in independent packages referred to as 'blocks'. The blocks are synced to all other nodes in the network. Blocks and user IDs cannot be easily proven to have been created by particular nodes (only inferred). Even if there is enough evidence to believe a particular node created a block, nodes still operate behind Tor or I2P and as such are not trivially known to be at a particular IP address.
+Onionr stores data in independent packages referred to as 'blocks'. The blocks are synced to all other nodes in the network. Blocks and user IDs cannot be easily proven to have been created by a particular user. Even if there is enough evidence to believe that a specific user created a block, nodes still operate behind Tor or I2P and as such cannot be trivially unmasked.
 
-Users are identified by ed25519 public keys, which can be used to sign blocks or send encrypted data.
+Users are identified by ed25519/curve25519 public keys, which can be used to sign blocks or send encrypted data.
 
 Onionr can be used for mail, as a social network, instant messenger, file sharing software, or for encrypted group discussion.
 
+The whitepaper (subject to change prior to alpha release) is available [here](docs/whitepaper.md).
+
 ![Tor stinks slide image](docs/tor-stinks-02.png)
 
 ## Main Features
@@ -34,17 +36,33 @@ Onionr can be used for mail, as a social network, instant messenger, file sharin
 * [X] 🌐 Fully p2p/decentralized, no trackers or other single points of failure
 * [X] 🔒 End to end encryption of user data
 * [X] 📢 Optional non-encrypted blocks, useful for blog posts or public file sharing
-* [X] 💻 Easy API system for integration to websites
+* [X] 💻 Easy HTTP API for integration to websites
 * [X] 🕵️ Metadata analysis resistance and anonymity
 * [X] 📡 Transport agnosticism (no internet required)
 
+## Software Suite
+
+Onionr ships with various application plugins ready for use out of the box:
+
+Currently usable:
+
+* Mail
+* Public anonymous chat
+* Simple webpage hosting - Will be greatly extended
+* File sharing (Work in progress)
+
+Not yet usable:
+
+* Instant messaging
+* Forum/BBS
+
 **Onionr API and functionality is subject to non-backwards compatible change during pre-alpha development**
 
 # Screenshots
 
 <img alt='Node statistics page screenshot' src='docs/onionr-1.png' width=600>
 
-Node statistics
+Home screen
 
 <img alt='Friend/contact manager screenshot' src='docs/onionr-2.png' width=600>
 
@@ -52,22 +70,31 @@ Friend/contact manager
 
 <img alt='Encrypted, metadata-masking mail application screenshot' src='docs/onionr-3.png' width=600>
 
-Encrypted, metadata-masking mail application.
+Encrypted, metadata-masking mail application. One of the first distributed mail systems to have basic forward secrecy.
+
+# Documentation
+
+More docs coming soon.
+
+* [Block specification](docs/specs/block-spec.md)
+* [HTTP API](docs/http-api.md)
 
 # Install and Run on Linux
 
 The following applies to Ubuntu Bionic. Other distros may have different package or command names.
 
-* Have python3.6+, python3-pip, Tor (daemon, not browser) installed (python3-dev recommended)
+`$ sudo apt install python3-pip python3-dev tor`
+
+* Have python3.6+, python3-pip, Tor (daemon, not browser) installed. python3-dev is recommended.
 * Clone the git repo: `$ git clone https://gitlab.com/beardog/onionr`
 * cd into install direction: `$ cd onionr/`
 * Install the Python dependencies ([virtualenv strongly recommended](https://virtualenv.pypa.io/en/stable/userguide/)): `$ pip3 install --require-hashes -r requirements.txt`
 
-(--require-hashes is intended to prevent exploitation via compromise of Pypi/CA certificates)
+(--require-hashes is intended to prevent exploitation via compromise of PyPi/CA certificates)
 
-## Help out
+# Help out
 
-Everyone is welcome to help out. Help is wanted for the following:
+Everyone is welcome to contribute. Help is wanted for the following:
 
 * Development (Get in touch first)
     * Creation of a shared lib for use from other languages and faster proof-of-work
@@ -75,26 +102,39 @@ Everyone is welcome to help out. Help is wanted for the following:
     * Windows and Mac support (already partially supported, testers needed)
     * General bug fixes and development of new features
 * Testing
+* Translations/localizations
 * UI/UX design
 * Running stable nodes
 * Security review/audit
 * Automatic I2P setup
 
-Contribute money:
+## Contribute money:
 
 Donating at least $5 gets you cool Onionr stickers. Get in touch if you want them.
 
-Bitcoin: [1onion55FXzm6h8KQw3zFw2igpHcV7LPq](bitcoin:1onion55FXzm6h8KQw3zFw2igpHcV7LPq) (Contact us for privacy coins like Monero)
+Bitcoin: [1onion55FXzm6h8KQw3zFw2igpHcV7LPq](bitcoin:1onion55FXzm6h8KQw3zFw2igpHcV7LPq) (Contact us for a unique address or for other coins)
+
+Monero: 4B5BA24d1P3R5aWEpkGY5TP7buJJcn2aSGBVRQCHhpiahxeB4aWsu15XwmuTjC6VF62NApZeJGTS248RMVECP8aW73Uj2ax
 
 USD (Card/Paypal): [Ko-Fi](https://www.ko-fi.com/beardogkf)
 
 Note: probably not tax deductible
 
-## Contact
+# Contact
 
-beardog [ at ] mailbox.org
+Email: beardog [ at ] mailbox.org
 
-## Disclaimer
+Onionr Mail: TRH763JURNY47QPBTTQ4LLPYCYQK6Q5YA33R6GANKZK5C5DKCIGQ
+
+## Disclaimers and legal
+
+No matter how good Onionr and other software gets, there will always be ways for clever or well-funded adversaries to break your security. 
+
+*Do not rely on Onionr or any other software to hold up if your life or liberty are at stake.*
+
+### Licenses and Branding
+
+Onionr is published under the GNU GPL v3 license, except for the logo.
 
 The Tor Project and I2P developers do not own, create, or endorse this project, and are not otherwise involved.
 
@@ -102,8 +142,10 @@ Tor is a trademark for the Tor Project. We do not own it.
 
 The 'open source badge' is by Maik Ellerbrock and is licensed under a Creative Commons Attribution 4.0 International License.
 
-## Logo
+## Onionr Logo
 
 The Onionr logo was created by [Anhar Ismail](https://github.com/anharismail) under the [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/). 
 
-If you modify and redistribute our code ("forking"), please use a different logo and project name to avoid confusion. Please do not use our logo in a way that makes it seem like we endorse you without permission.
+If you modify and redistribute our code ("forking"), please use a different logo and project name to avoid confusion. Please do not use the project name or logo in a way that makes it seem like we endorse you without our permission.
+
+![node web illustration](docs/onionr-web.png)

SECURITY.md

@@ -0,0 +1,39 @@
+# Security Policy
+
+We welcome responsible and constructive security review.
+
+# Scope
+
+The Onionr software and any nodes you control are within scope.
+
+Avoid social engineering, volume-based denial of service and disrupting or harming the Onionr network. Do not attempt to exploit any machines/servers you do not own or otherwise have permission to do so.
+
+The following exploits are of particular interest:
+
+* Arbitrary code execution
+* API authentication bypass (such as accessing local API from public interface)
+* Deanonymization:
+  * Easily associating public keys with server addresses
+  * Discovering true server IPs when behind Tor/I2P (aside from Tor/i2p-level attacks)
+  * Easily discovering which nodes are the block creator
+* XSS, CSRF, clickjacking, DNS rebinding
+* Timing attacks against the local http server ([see blog post](https://www.chaoswebs.net/blog/timebleed-breaking-privacy-with-a-simple-timing-attack.html))
+* Discovering direct connection servers as a non participant.
+* Cryptography/protocol issues
+* Denying nodes access to the network by segmenting them out with Sybil nodes
+
+We do not consider non-network based same-machine attacks to be very significant, but we are still willing to listen.
+
+# Rewards
+
+Onionr is a student-owned hobby project, resources are not available for large rewards.
+
+Stickers or other small rewards are available. We reserve the right to refuse rewards for any reason.
+
+Public recognition can be given upon request.
+
+# Contact
+
+Email: beardog [ at ] mailbox.org
+
+PGP (optional): F61A 4DBB 0B3D F172 1F65  0EDF 0D41 4D0F E405 B63B

docs/README.md

@@ -0,0 +1,19 @@
+# Onionr Documentation
+
+The Onionr [whitepaper](whitepaper.md) is the best place to start both for users and developers.
+
+## User Documentation
+
+* [Installation](usage/install.md)
+* [First steps](usage/firststeps.md)
+* [Using Onionr Mail](usage/mail.md)
+* [Using Onionr web pages](usage/pages.md)
+* [Staying safe/anonymous](usage/safety.md)
+
+## Developer Documentation
+
+* [Development environment setup](dev/setup.md)
+* [Technical overview](dev/overview.md)
+* [Project layout](dev/layout.md)
+* [Plugin development guide](dev/plugins.md)
+* [Testing](dev/testing.md)

docs/api.md

@@ -1,2 +0,0 @@
-HTTP API
-TODO

docs/dev/http-api.md

@@ -0,0 +1,118 @@
+# Onionr HTTP API
+
+All HTTP interfaces in the Onionr reference client use the [Flask](http://flask.pocoo.org/) web framework with the [gevent](http://www.gevent.org/) WSGI server.
+
+## Client & Public difference
+
+The client API server is a locked down interface intended for authenticated local communication. 
+
+The public API server is available only remotely from Tor & I2P. It is the interface in which peers use to communicate with one another.
+
+# Client API
+
+Please note: endpoints that simply provide static web app files are not documented here.
+
+* /serviceactive/pubkey
+    - Methods: GET
+    - Returns true or false based on if a given public key has an active direct connection service.
+* /queueResponseAdd/key (DEPRECATED)
+    - Methods: POST
+    - Accepts form key 'data' to set queue response information from a plugin
+    - Returns success if no error occurs
+* /queueResponse/key (DEPRECATED)
+    - Methods: GET
+    - Returns the queue response for a key. Returns failure with a 404 code if a code is not set.
+* /ping
+    - Methods: GET
+    - Returns "pong!"
+* /getblocksbytype/type
+    - Methods: GET
+    - Returns a list of stored blocks by a given type
+* /getblockbody/hash
+    - Methods: GET
+    - Returns the main data section of a block
+* /getblockdata/hash
+    - Methods: GET
+    - Returns the entire data contents of a block, including metadata.
+* /getblockheader/hash
+    - Methods: GET
+    - Returns the header (metadata section) of a block.
+* /hitcount
+    - Methods: GET
+    - Return the amount of requests the public api server has received this session
+* /lastconnect
+    - Methods: GET
+    - Returns the epoch timestamp of when the last incoming connection to the public API server was logged
+* /site/hash
+    - Methods: GET
+    - Returns HTML content out of a block
+* /waitforshare/hash
+    - Methods: POST
+    - Prevents the public API server from listing or sharing a block until it has been uploaded to at least 1 peer.
+* /shutdown
+    - Methods: GET
+    - Shutdown Onionr. You should probably use /shutdownclean instead.
+* /shutdownclean
+    - Methods: GET
+    - Tells the communicator daemon to shutdown Onionr. Slower but cleaner.
+* /getstats
+    - Methods: GET
+    - Returns some JSON serialized statistics
+* /getuptime
+    - Methods: GET
+    - Returns uptime in seconds
+* /getActivePubkey
+    - Methods: GET
+    - Returns the current active public key in base32 format
+* /getHumanReadable/pubkey
+    - Methods: GET
+    - Echos the specified public key in mnemonic format
+* /insertblock
+    - Methods: POST
+    - Accepts JSON data for creating a new block. 'message' contains the block data, 'to' specifies the peer's public key to encrypt the data to, 'sign' is a boolean for signing the message.
+
+# Public API
+
+v0
+
+* /
+    - Methods: GET
+    - Returns a basic HTML informational banner describing Onionr.
+* /getblocklist
+    - Methods: GET
+    - URI Parameters:
+        - date: unix epoch timestamp for offset
+    - Returns a list of block hashes stored on the node since an offset (all blocks if no timestamp is specified)
+* /getdata/block-hash
+    - Methods: GET
+    - Returns data for a block based on a provided hash
+* /www/file-path
+    - Methods: GET
+    - Returns file data. Intended for manually sharing file data directly from an Onionr node.
+* /ping
+    - Methods: GET
+    - Returns 'pong!'
+* /pex
+    - Methods: GET
+    - Returns a list of peer addresses reached within recent time
+* /announce
+    - Methods: POST
+    - Accepts form data for 'node' (valid node address) and 'random' which is a nonce when hashed (blake2b_256) in the format `hash(peerAddress+serverAddress+nonce)`, begins with at least 5 zeros.
+    - Returns 200 with 'Success' if no error occurs. If the post is invalid, 'failure' with code 406 is returned.
+* /upload
+    - Methods: POST
+    - Accepts form data for 'block' as a 'file' upload.
+    - Returns 200 with 'success' if no error occurs. If the block cannot be accepted, 'failure' with 400 is returned.
+
+# Direct Connection API
+
+These are constant endpoints available on direct connection servers. Plugin endpoints for direct connections are not documented here.
+
+* /ping
+    - Methods: GET
+    - Returns 200 with 'pong!'
+
+* /close
+    - Methods: GET
+    - Kills the direct connection server, destroying the onion address.
+    - Returns 200 with 'goodbye'

docs/dev/specs/block-spec.md

@@ -0,0 +1,63 @@
+# Onionr Block Spec v1.1.0
+
+# Block Description
+
+Onionr Blocks are the primary means of sharing information in Onionr. Blocks are identified by a single hash value of their entire contents, using SHA3_256.
+
+Blocks contain a JSON metadata section followed by a line break, with the main data section comprising the rest.
+
+In the future, the specification will likely be updated to use flags and MessagePack instead of JSON with english keys.
+
+# Encryption and Signatures
+
+Onionr blocks may be encrypted or signed. In the reference client, this is done with libsodium, for both asymmetric and symmetric encryption.
+
+Unlike many similar projects, blocks may completely be in plaintext, making Onionr suitable for sharing information publicly.
+
+# Metadata Section
+
+The metadata section has the following fields. If a block contains any other field, it must be considered invalid. All metadata fields are technically optional, but many are useful and essentially necessary for most use cases.
+
+## meta
+
+Max byte size: 1000
+
+Meta is a string field which can contain arbitrary sub fields. It is intended for applications and plugins to use it for arbitrary metadata information. In the reference client, if the data section is encrypted or signed, the meta section also is.
+
+Common meta fields, such as 'type' are used by the reference Onionr client to describe the type of a block.
+
+## sig
+
+Max byte size: 200
+
+Sig is a field for storing public key signatures of the block, typically ed25519. In the reference client, this field is a base64 encoded signature of the meta field combined with the block data. (**Therefore, information outside of the meta and data fields cannot be trusted to be placed there by the signer, although it can still be assured that the particular block has not been modified.**)
+
+Note: the max field size is larger than a EdDSA signature (which is what is typically used) in order to allow other primitives for signing in alternative implementations or future versions.
+
+## signer
+
+Max byte size: 200
+
+Signer is a field for specifying the public key which signed the block. In the reference client this is a base64 encoded ed25519 public key.
+
+## time
+
+Max byte size: 10
+
+Time is an integer field for specifying the time of which a block was created. The trustworthiness of this field is based on one's trust of the block creator, however blocks with a time field set in the future (past a reasonable clock skew) are thrown out by the reference client.
+
+## expire
+
+Max byte size: 10
+
+Expire is an integer field for specifying the time of which the block creator has indicated that the block should be deleted. The purpose of this is for voluntarily freeing the burden of unwanted blocks on the Onionr network, rather than security/privacy (since blocks could be trivially kept past expiration). Regardless, the reference client deletes blocks after a preset time if the expire field is either not set or longer than the preset time.
+
+## pow
+
+Max byte size: 1000
+
+Pow is a field for placing the nonce found to make a block meet a target proof of work. In theory, a block could meet a target without a random token in this field.
+
+## encryptType
+
+encryptType is a field to specify the mode of encryption for a block. The values supported by Onionr are 'asym' and 'sym'.

docs/usage/firststeps.md

@@ -0,0 +1,7 @@
+# Onionr First Steps
+
+After installing Onionr, there are several things to do:
+
+1. Setup a [deterministic address](usage/deterministic.md) (optional)
+2. Add friends' ids
+3. Publish your id

docs/usage/install.md

@@ -0,0 +1,13 @@
+# Onionr Installation
+
+The following steps work broadly speaking for Windows, Mac, and Linux.
+
+1. Verify python3.6+ is installed: if its not see https://www.python.org/downloads/
+
+2. Verify Tor is installed (does not need to be running, binary can be put into system path or Onionr directory)
+
+3. [Optional but recommended]: setup virtual environment using [virtualenv](https://virtualenv.pypa.io/en/latest/), activate the virtual environment
+
+4. Clone Onionr: git clone https://gitlab.com/beardog/onionr
+
+5. Install the Python module dependencies: pip3 install --require-hashes -r requirements.txt

docs/whitepaper.md

@@ -39,11 +39,11 @@ When designing Onionr we had these main goals in mind:
 
 At its core, Onionr is merely a description for storing data in self-verifying packages ("blocks"). These blocks can be encrypted to a user (or for one's self), encrypted symmetrically, or not at all. Blocks can be signed by their creator, but regardless, they are self-verifying due to being identified by a sha3-256 hash value; once a block is created, it cannot be modified.
 
-Onionr exchanges a list of blocks between all nodes. By default, all nodes download and share all other blocks, however this is configurable. Blocks do not rely on any particular order of receipt or transport mechanism.
+Onionr exchanges a list of blocks between all nodes. By default, all nodes download and share all other blocks, however, this is configurable. Blocks do not rely on any particular order of receipt or transport mechanism.
 
 ## User IDs
 
-User IDs are simply Ed25519 public keys. They are represented in Base32 format, or encoded using the [PGP Word List](https://en.wikipedia.org/wiki/PGP_word_list).
+User IDs are simply Ed25519 public keys. They are represented in Base32 format or encoded using the [PGP Word List](https://en.wikipedia.org/wiki/PGP_word_list).
 
 Public keys can be generated deterministically with a password using a key derivation function (Argon2id). This password can be shared between many users in order to share data anonymously among a group, using only 1 password. This is useful in some cases, but is risky, as if one user causes the key to be compromised and does not notify the group or revoke the key, there is no way to know.
 
@@ -53,7 +53,7 @@ Although Onionr is transport agnostic, the only supported transports in the refe
 
 ### Node Profiling
 
-To mitigate maliciously slow or unreliable nodes, Onionr builds a profile on nodes it connects to. Nodes are assigned a score, which raises based on the amount of successful block transfers, speed, and reliability of a node, and reduces the score based on how unreliable a node is. If a node is unreachable for over 24 hours after contact, it is forgotten. Onionr can also prioritize connection to 'friend' nodes.
+To mitigate maliciously slow or unreliable nodes, Onionr builds a profile on nodes it connects to. Nodes are assigned a score, which raises based on the number of successful block transfers, speed, and reliability of a node, and reduces the score based on how unreliable a node is. If a node is unreachable for over 24 hours after contact, it is forgotten. Onionr can also prioritize connections to 'friend' nodes.
 
 ## Block Format
 
@@ -65,7 +65,7 @@ Optionally, a random token can be inserted into the metadata for use in Proof of
 
 ### Block Encryption
 
-For encryption, Onionr uses ephemeral Curve25519 keys for key exchange and XSalsa20-Poly1305 as a symmetric cipher, or optionally using only XSalsa20-Poly1305 with a pre-shared key.
+For encryption, Onionr uses ephemeral Curve25519 keys for key exchange and XSalsa20-Poly1305 as a symmetric cipher or optionally using only XSalsa20-Poly1305 with a pre-shared key.
 
 Regardless of encryption, blocks can be signed internally using Ed25519.
 
@@ -91,7 +91,7 @@ In addition, randomness beacons such as the one operated by [NIST](https://beaco
 
 # Direct Connections
 
-We propose a method of using Onionr's block sync system to enable direct connections between peers by having one peer request to connect to another using the peer's public key. Since the request is within a standard block, proof of work must be used to request connection. If the requested peer is available and wishes to accept the connection, Onionr will generate a temporary .onion address for the other peer to connect to. Alternatively, a reverse connection may be formed, which is faster to establish but requires a message brokering system instead of a standard socket.
+We propose a method of using Onionr's block sync system to enable direct connections between peers by having one peer request to connect to another using the peer's public key. Since the request is within a standard block, proof of work must be used to request a connection. If the requested peer is available and wishes to accept the connection, Onionr will generate a temporary .onion address for the other peer to connect to. Alternatively, a reverse connection may be formed, which is faster to establish but requires a message brokering system instead of a standard socket.
 
 The benefits of such a system are increased privacy, and the ability to anonymously communicate from multiple devices at once. In a traditional onion service, one's online status can be monitored and more easily correlated.
 
@@ -119,7 +119,9 @@ We seek to protect the following information:
 * Physical location/IP address of nodes on the network
 * All block data from tampering
 
-### Data we cannot or do not protect
+### Unprotected Data
+
+Onionr does not protect the following:
 
 * Data specifically inserted as plaintext is available to the public
 * The public key of signed plaintext blocks
@@ -133,12 +135,6 @@ We assume that Tor onion services (v3) and I2P services cannot be trivially dean
 
 Once quantum safe algorithms are more mature and have decent high level libraries, they will be deployed.
 
-# Comparisons to other P2P software
-
-Since Onionr is far from the first to implement many of these ideas (on their own), this section compares Onionr to other networks, using points we consider to be the most important.
-
-![network comparison image](network-comparison.png)
-
 # Conclusion
 
-If successful, Onionr will be a complete decentralized platform for anonymous computing, complete with limited metadata exposure, both node and user anonymity, and spam prevention
+If successful, Onionr will be a complete decentralized platform for anonymous computing, complete with limited metadata exposure, both node and user anonymity, and spam prevention

onionr/api.py

@@ -1,602 +0,0 @@
-'''
-    Onionr - P2P Anonymous Storage Network
-
-    This file handles all incoming http requests to the client, using Flask
-'''
-'''
-    This program is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation, either version 3 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License
-    along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
-from gevent.pywsgi import WSGIServer, WSGIHandler
-from gevent import Timeout
-import flask, cgi, uuid
-from flask import request, Response, abort, send_from_directory
-import sys, random, threading, hmac, base64, time, os, json, socket
-import core
-from onionrblockapi import Block
-import onionrutils, onionrexceptions, onionrcrypto, blockimporter, onionrevents as events, logger, config
-import httpapi
-from httpapi import friendsapi, simplecache
-import onionr
-
-class FDSafeHandler(WSGIHandler):
-    '''Our WSGI handler. Doesn't do much non-default except timeouts'''
-    def handle(self):
-       timeout = Timeout(60, exception=Exception)
-       timeout.start()
-       try:
-           WSGIHandler.handle(self)
-       except Timeout as ex:
-           raise
-
-def setBindIP(filePath):
-    '''Set a random localhost IP to a specified file (intended for private or public API localhost IPs)'''
-    if config.get('general.random_bind_ip', True):
-        hostOctets = [str(127), str(random.randint(0x02, 0xFF)), str(random.randint(0x02, 0xFF)), str(random.randint(0x02, 0xFF))]
-        data = '.'.join(hostOctets)
-        # Try to bind IP. Some platforms like Mac block non normal 127.x.x.x
-        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        try:
-            s.bind((data, 0))
-        except OSError:
-            # if mac/non-bindable, show warning and default to 127.0.0.1
-            logger.warn('Your platform appears to not support random local host addresses 127.x.x.x. Falling back to 127.0.0.1.')
-            data = '127.0.0.1'
-        s.close()
-    else:
-        data = '127.0.0.1'
-    with open(filePath, 'w') as bindFile:
-        bindFile.write(data)
-    return data
-
-class PublicAPI:
-    '''
-        The new client api server, isolated from the public api
-    '''
-    def __init__(self, clientAPI):
-        assert isinstance(clientAPI, API)
-        app = flask.Flask('PublicAPI')
-        self.i2pEnabled = config.get('i2p.host', False)
-        self.hideBlocks = [] # Blocks to be denied sharing
-        self.host = setBindIP(clientAPI._core.publicApiHostFile)
-        self.torAdder = clientAPI._core.hsAddress
-        self.i2pAdder = clientAPI._core.i2pAddress
-        self.bindPort = config.get('client.public.port')
-        self.lastRequest = 0
-        logger.info('Running public api on %s:%s' % (self.host, self.bindPort))
-
-        @app.before_request
-        def validateRequest():
-            '''Validate request has the correct hostname'''
-            # If high security level, deny requests to public (HS should be disabled anyway for Tor, but might not be for I2P)
-            if config.get('general.security_level', default=0) > 0:
-                abort(403)
-            if type(self.torAdder) is None and type(self.i2pAdder) is None:
-                # abort if our hs addresses are not known
-                abort(403)
-            if request.host not in (self.i2pAdder, self.torAdder):
-                # Disallow connection if wrong HTTP hostname, in order to prevent DNS rebinding attacks
-                abort(403)
-
-        @app.after_request
-        def sendHeaders(resp):
-            '''Send api, access control headers'''
-            resp.headers['Date'] = 'Thu, 1 Jan 1970 00:00:00 GMT' # Clock info is probably useful to attackers. Set to unix epoch, since we can't fully remove the header.
-            # CSP to prevent XSS. Mainly for client side attacks (if hostname protection could somehow be bypassed)
-            resp.headers["Content-Security-Policy"] =  "default-src 'none'; script-src 'none'; object-src 'none'; style-src data: 'unsafe-inline'; img-src data:; media-src 'none'; frame-src 'none'; font-src 'none'; connect-src 'none'"
-            # Prevent click jacking
-            resp.headers['X-Frame-Options'] = 'deny'
-            # No sniff is possibly not needed
-            resp.headers['X-Content-Type-Options'] = "nosniff"
-            # Network API version
-            resp.headers['X-API'] = onionr.API_VERSION
-            # Close connections to limit FD use
-            resp.headers['Connection'] = "close"
-            self.lastRequest = clientAPI._core._utils.getRoundedEpoch(roundS=5)
-            return resp
-
-        @app.route('/')
-        def banner():
-            # Display a bit of information to people who visit a node address in their browser
-            try:
-                with open('static-data/index.html', 'r') as html:
-                    resp = Response(html.read(), mimetype='text/html')
-            except FileNotFoundError:
-                resp = Response("")
-            return resp
-
-        @app.route('/getblocklist')
-        def getBlockList():
-            # Provide a list of our blocks, with a date offset
-            dateAdjust = request.args.get('date')
-            bList = clientAPI._core.getBlockList(dateRec=dateAdjust)
-            for b in self.hideBlocks:
-                if b in bList:
-                    # Don't share blocks we created if they haven't been *uploaded* yet, makes it harder to find who created a block
-                    bList.remove(b)
-            return Response('\n'.join(bList))
-
-        @app.route('/getdata/<name>')
-        def getBlockData(name):
-            # Share data for a block if we have it
-            resp = ''
-            data = name
-            if clientAPI._utils.validateHash(data):
-                if data not in self.hideBlocks:
-                    if data in clientAPI._core.getBlockList():
-                        block = clientAPI.getBlockData(data, raw=True)
-                        try:
-                            block = block.encode()
-                        except AttributeError:
-                            abort(404)
-                        block = clientAPI._core._utils.strToBytes(block)
-                        resp = block
-                        #resp = base64.b64encode(block).decode()
-            if len(resp) == 0:
-                abort(404)
-                resp = ""
-            return Response(resp, mimetype='application/octet-stream')
-
-        @app.route('/www/<path:path>')
-        def wwwPublic(path):
-            # A way to share files directly over your .onion
-            if not config.get("www.public.run", True):
-                abort(403)
-            return send_from_directory(config.get('www.public.path', 'static-data/www/public/'), path)
-
-        @app.route('/ping')
-        def ping():
-            # Endpoint to test if nodes are up
-            return Response("pong!")
-
-        @app.route('/pex')
-        def peerExchange():
-            response = ','.join(clientAPI._core.listAdders(recent=3600))
-            if len(response) == 0:
-                response = ''
-            return Response(response)
-        
-        @app.route('/announce', methods=['post'])
-        def acceptAnnounce():
-            resp = 'failure'
-            powHash = ''
-            randomData = ''
-            newNode = ''
-            ourAdder = clientAPI._core.hsAddress.encode()
-            try:
-                newNode = request.form['node'].encode()
-            except KeyError:
-                logger.warn('No node specified for upload')
-                pass
-            else:
-                try:
-                    randomData = request.form['random']
-                    randomData = base64.b64decode(randomData)
-                except KeyError:
-                    logger.warn('No random data specified for upload')
-                else:
-                    nodes = newNode + clientAPI._core.hsAddress.encode()
-                    nodes = clientAPI._core._crypto.blake2bHash(nodes)
-                    powHash = clientAPI._core._crypto.blake2bHash(randomData + nodes)
-                    try:
-                        powHash = powHash.decode()
-                    except AttributeError:
-                        pass
-                    if powHash.startswith('0000'):
-                        newNode = clientAPI._core._utils.bytesToStr(newNode)
-                        if clientAPI._core._utils.validateID(newNode) and not newNode in clientAPI._core.onionrInst.communicatorInst.newPeers:
-                            clientAPI._core.onionrInst.communicatorInst.newPeers.append(newNode)
-                            resp = 'Success'
-                    else:
-                        logger.warn(newNode.decode() + ' failed to meet POW: ' + powHash)
-            resp = Response(resp)
-            return resp
-
-        @app.route('/upload', methods=['post'])
-        def upload():
-            '''Accept file uploads. In the future this will be done more often than on creation 
-            to speed up block sync
-            '''
-            resp = 'failure'
-            try:
-                data = request.form['block']
-            except KeyError:
-                logger.warn('No block specified for upload')
-                pass
-            else:
-                if sys.getsizeof(data) < 100000000:
-                    try:
-                        if blockimporter.importBlockFromData(data, clientAPI._core):
-                            resp = 'success'
-                        else:
-                            logger.warn('Error encountered importing uploaded block')
-                    except onionrexceptions.BlacklistedBlock:
-                        logger.debug('uploaded block is blacklisted')
-                        pass
-            if resp == 'failure':
-                abort(400)
-            resp = Response(resp)
-            return resp
-
-        # Set instances, then startup our public api server
-        clientAPI.setPublicAPIInstance(self)
-        while self.torAdder == '':
-            clientAPI._core.refreshFirstStartVars()
-            self.torAdder = clientAPI._core.hsAddress
-            time.sleep(0.1)
-        self.httpServer = WSGIServer((self.host, self.bindPort), app, log=None, handler_class=FDSafeHandler)
-        self.httpServer.serve_forever()
-
-class API:
-    '''
-        Client HTTP api
-    '''
-
-    callbacks = {'public' : {}, 'private' : {}}
-
-    def __init__(self, onionrInst, debug, API_VERSION):
-        '''
-            Initialize the api server, preping variables for later use
-
-            This initilization defines all of the API entry points and handlers for the endpoints and errors
-            This also saves the used host (random localhost IP address) to the data folder in host.txt
-        '''
-
-        self.debug = debug
-        self._core = onionrInst.onionrCore
-        self.startTime = self._core._utils.getEpoch()
-        self._crypto = onionrcrypto.OnionrCrypto(self._core)
-        self._utils = onionrutils.OnionrUtils(self._core)
-        app = flask.Flask(__name__)
-        bindPort = int(config.get('client.client.port', 59496))
-        self.bindPort = bindPort
-
-        # Be extremely mindful of this. These are endpoints available without a password
-        self.whitelistEndpoints = ('site', 'www', 'onionrhome', 'board', 'boardContent', 'sharedContent', 'mail', 'mailindex', 'friends', 'friendsindex')
-
-        self.clientToken = config.get('client.webpassword')
-        self.timeBypassToken = base64.b16encode(os.urandom(32)).decode()
-
-        self.publicAPI = None # gets set when the thread calls our setter... bad hack but kinda necessary with flask
-        #threading.Thread(target=PublicAPI, args=(self,)).start()
-        self.host = setBindIP(self._core.privateApiHostFile)
-        logger.info('Running api on %s:%s' % (self.host, self.bindPort))
-        self.httpServer = ''
-
-        self.pluginResponses = {} # Responses for plugin endpoints
-        self.queueResponse = {}
-        onionrInst.setClientAPIInst(self)
-        app.register_blueprint(friendsapi.friends)
-        app.register_blueprint(simplecache.simplecache)
-        httpapi.load_plugin_blueprints(app)
-
-        @app.before_request
-        def validateRequest():
-            '''Validate request has set password and is the correct hostname'''
-            # For the purpose of preventing DNS rebinding attacks
-            if request.host != '%s:%s' % (self.host, self.bindPort):
-                abort(403)
-            if request.endpoint in self.whitelistEndpoints:
-                return
-            try:
-                if not hmac.compare_digest(request.headers['token'], self.clientToken):
-                    if not hmac.compare_digest(request.form['token'], self.clientToken):
-                        abort(403)
-            except KeyError:
-                if not hmac.compare_digest(request.form['token'], self.clientToken):
-                    abort(403)
-
-        @app.after_request
-        def afterReq(resp):
-            # Security headers
-            if request.endpoint == 'site':
-                resp.headers['Content-Security-Policy'] = "default-src 'none'; style-src data: 'unsafe-inline'; img-src data:"
-            else:
-                resp.headers['Content-Security-Policy'] = "default-src 'none'; script-src 'self'; object-src 'none'; style-src 'self'; img-src 'self'; media-src 'none'; frame-src 'none'; font-src 'none'; connect-src 'self'"
-            resp.headers['X-Frame-Options'] = 'deny'
-            resp.headers['X-Content-Type-Options'] = "nosniff"
-            resp.headers['Server'] = ''
-            resp.headers['Date'] = 'Thu, 1 Jan 1970 00:00:00 GMT' # Clock info is probably useful to attackers. Set to unix epoch.
-            resp.headers['Connection'] = "close"
-            return resp
-
-        @app.route('/board/', endpoint='board')
-        def loadBoard():
-            return send_from_directory('static-data/www/board/', "index.html")
-
-        @app.route('/mail/<path:path>', endpoint='mail')
-        def loadMail(path):
-            return send_from_directory('static-data/www/mail/', path)
-        @app.route('/mail/', endpoint='mailindex')
-        def loadMailIndex():
-            return send_from_directory('static-data/www/mail/', 'index.html')
-        
-        @app.route('/friends/<path:path>', endpoint='friends')
-        def loadContacts(path):
-            return send_from_directory('static-data/www/friends/', path)
-
-        @app.route('/friends/', endpoint='friendsindex')
-        def loadContacts():
-            return send_from_directory('static-data/www/friends/', 'index.html')
-
-        @app.route('/board/<path:path>', endpoint='boardContent')
-        def boardContent(path):
-            return send_from_directory('static-data/www/board/', path)
-        @app.route('/shared/<path:path>', endpoint='sharedContent')
-        def sharedContent(path):
-            return send_from_directory('static-data/www/shared/', path)
-
-        @app.route('/www/<path:path>', endpoint='www')
-        def wwwPublic(path):
-            if not config.get("www.private.run", True):
-                abort(403)
-            return send_from_directory(config.get('www.private.path', 'static-data/www/private/'), path)
-
-        @app.route('/queueResponseAdd/<name>', methods=['post'])
-        def queueResponseAdd(name):
-            # Responses from the daemon. TODO: change to direct var access instead of http endpoint
-            self.queueResponse[name] = request.form['data']
-            return Response('success')
-        
-        @app.route('/queueResponse/<name>')
-        def queueResponse(name):
-            # Fetch a daemon queue response
-            resp = 'failure'
-            try:
-                resp = self.queueResponse[name]
-            except KeyError:
-                pass
-            else:
-                del self.queueResponse[name]
-            return Response(resp)
-            
-        @app.route('/ping')
-        def ping():
-            # Used to check if client api is working
-            return Response("pong!")
-
-        @app.route('/', endpoint='onionrhome')
-        def hello():
-            # ui home
-            return send_from_directory('static-data/www/private/', 'index.html')
-        
-        @app.route('/getblocksbytype/<name>')
-        def getBlocksByType(name):
-            blocks = self._core.getBlocksByType(name)
-            return Response(','.join(blocks))
-        
-        @app.route('/getblockbody/<name>')
-        def getBlockBodyData(name):
-            resp = ''
-            if self._core._utils.validateHash(name):
-                try:
-                    resp = Block(name, decrypt=True).bcontent
-                    #resp =  cgi.escape(Block(name, decrypt=True).bcontent, quote=True)
-                except TypeError:
-                    pass
-            else:
-                abort(404)
-            return Response(resp)
-        
-        @app.route('/getblockdata/<name>')
-        def getData(name):
-            resp = ""
-            if self._core._utils.validateHash(name):
-                if name in self._core.getBlockList():
-                    try:
-                        resp = self.getBlockData(name, decrypt=True)
-                    except ValueError:
-                        pass
-                else:
-                    abort(404)
-            else:
-                abort(404)
-            return Response(resp)
-
-        @app.route('/getblockheader/<name>')
-        def getBlockHeader(name):
-            resp = self.getBlockData(name, decrypt=True, headerOnly=True)
-            return Response(resp)
-
-        @app.route('/lastconnect')
-        def lastConnect():
-            return Response(str(self.publicAPI.lastRequest))
-
-        @app.route('/site/<name>', endpoint='site')
-        def site(name):
-            bHash = name
-            resp = 'Not Found'
-            if self._core._utils.validateHash(bHash):
-                try:
-                    resp = Block(bHash).bcontent
-                except onionrexceptions.NoDataAvailable:
-                    abort(404)
-                except TypeError:
-                    pass
-                try:
-                    resp = base64.b64decode(resp)
-                except:
-                    pass
-            if resp == 'Not Found' or not resp:
-                abort(404)
-            return Response(resp)
-
-        @app.route('/waitforshare/<name>', methods=['post'])
-        def waitforshare(name):
-            '''Used to prevent the **public** api from sharing blocks we just created'''
-            assert name.isalnum()
-            if name in self.publicAPI.hideBlocks:
-                self.publicAPI.hideBlocks.remove(name)
-                return Response("removed")
-            else:
-                self.publicAPI.hideBlocks.append(name)
-                return Response("added")
-
-        @app.route('/shutdown')
-        def shutdown():
-            try:
-                self.publicAPI.httpServer.stop()
-                self.httpServer.stop()
-            except AttributeError:
-                pass
-            return Response("bye")
-
-        @app.route('/shutdownclean')
-        def shutdownClean():
-            # good for calling from other clients
-            self._core.daemonQueueAdd('shutdown')
-            return Response("bye")
-        
-        @app.route('/getstats')
-        def getStats():
-            # returns node stats
-            #return Response("disabled")
-            while True:
-                try:    
-                    return Response(self._core.serializer.getStats())
-                except AttributeError:
-                    pass
-        
-        @app.route('/getuptime')
-        def showUptime():
-            return Response(str(self.getUptime()))
-        
-        @app.route('/getActivePubkey')
-        def getActivePubkey():
-            return Response(self._core._crypto.pubKey)
-
-        @app.route('/getHumanReadable/<name>')
-        def getHumanReadable(name):
-            return Response(self._core._utils.getHumanReadableID(name))
-
-        @app.route('/insertblock', methods=['POST'])
-        def insertBlock():
-            encrypt = False
-            bData = request.get_json(force=True)
-            message = bData['message']
-            subject = 'temp'
-            encryptType = ''
-            sign = True
-            meta = {}
-            to = ''
-            try:
-                if bData['encrypt']:
-                    to = bData['to']
-                    encrypt = True
-                    encryptType = 'asym'
-            except KeyError:
-                pass
-            try:
-                if not bData['sign']:
-                    sign = False
-            except KeyError:
-                pass
-            try:
-                bType = bData['type']
-            except KeyError:
-                bType = 'bin'
-            try:
-                meta = json.loads(bData['meta'])
-            except KeyError:
-                pass
-            threading.Thread(target=self._core.insertBlock, args=(message,), kwargs={'header': bType, 'encryptType': encryptType, 'sign':sign, 'asymPeer': to, 'meta': meta}).start()
-            return Response('success')
-        
-        @app.route('/apipoints/<path:subpath>', methods=['POST', 'GET'])
-        def pluginEndpoints(subpath=''):
-            '''Send data to plugins'''
-            # TODO have a variable for the plugin to set data to that we can use for the response
-            pluginResponseCode = str(uuid.uuid4())
-            resp = 'success'
-            responseTimeout = 20
-            startTime = self._core._utils.getEpoch()
-            postData = {}
-            if request.method == 'POST':
-                postData = request.form['postData']
-            if len(subpath) > 1:
-                data = subpath.split('/')
-                if len(data) > 1:
-                    plName = data[0]
-                    events.event('pluginRequest', {'name': plName, 'path': subpath, 'pluginResponse': pluginResponseCode, 'postData': postData}, onionr=onionrInst)
-                    while True:
-                        try:
-                            resp = self.pluginResponses[pluginResponseCode]
-                        except KeyError:
-                            time.sleep(0.2)
-                            if self._core._utils.getEpoch() - startTime > responseTimeout:
-                                abort(504)
-                                break
-                        else:
-                            break
-            else:
-                abort(404)
-            return Response(resp)
-
-        self.httpServer = WSGIServer((self.host, bindPort), app, log=None, handler_class=FDSafeHandler)
-        self.httpServer.serve_forever()
-
-    def setPublicAPIInstance(self, inst):
-        assert isinstance(inst, PublicAPI)
-        self.publicAPI = inst
-
-    def validateToken(self, token):
-        '''
-            Validate that the client token matches the given token. Used to prevent CSRF and data exfiltration
-        '''
-        if len(self.clientToken) == 0:
-            logger.error("client password needs to be set")
-            return False
-        try:
-            if not hmac.compare_digest(self.clientToken, token):
-                return False
-            else:
-                return True
-        except TypeError:
-            return False
-
-    def getUptime(self):
-        while True:
-            try:
-                return self._utils.getEpoch - startTime
-            except AttributeError:
-                # Don't error on race condition with startup
-                pass
-    
-    def getBlockData(self, bHash, decrypt=False, raw=False, headerOnly=False):
-        assert self._core._utils.validateHash(bHash)
-        bl = Block(bHash, core=self._core)
-        if decrypt:
-            bl.decrypt()
-            if bl.isEncrypted and not bl.decrypted:
-                raise ValueError
-
-        if not raw:
-            if not headerOnly:
-                retData = {'meta':bl.bheader, 'metadata': bl.bmetadata, 'content': bl.bcontent}
-                for x in list(retData.keys()):
-                    try:
-                        retData[x] = retData[x].decode()
-                    except AttributeError:
-                        pass
-            else:
-                validSig = False
-                signer = self._core._utils.bytesToStr(bl.signer)
-                #print(signer, bl.isSigned(), self._core._utils.validatePubKey(signer), bl.isSigner(signer))
-                if bl.isSigned() and self._core._utils.validatePubKey(signer) and bl.isSigner(signer):
-                    validSig = True                    
-                bl.bheader['validSig'] = validSig
-                bl.bheader['meta'] = ''
-                retData = {'meta': bl.bheader, 'metadata': bl.bmetadata}
-            return json.dumps(retData)
-        else:
-            return bl.raw

onionr/apiservers/__init__.py

@@ -0,0 +1,3 @@
+from . import public, private
+PublicAPI = public.PublicAPI
+ClientAPI = private.PrivateAPI

onionr/apiservers/private/__init__.py

@@ -0,0 +1,96 @@
+'''
+    Onionr - Private P2P Communication
+
+    This file handles all incoming http requests to the client, using Flask
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import base64, os
+import flask
+from gevent.pywsgi import WSGIServer
+import logger
+from onionrutils import epoch
+import httpapi
+from . import register_private_blueprints
+class PrivateAPI:
+    '''
+        Client HTTP api
+    '''
+
+    callbacks = {'public' : {}, 'private' : {}}
+
+    def __init__(self, onionrInst, debug, API_VERSION):
+        '''
+            Initialize the api server, preping variables for later use
+
+            This initialization defines all of the API entry points and handlers for the endpoints and errors
+            This also saves the used host (random localhost IP address) to the data folder in host.txt
+        '''
+        config = onionrInst.config
+        self.config = config
+        self.debug = debug
+        self._core = onionrInst.onionrCore
+        self.startTime = epoch.get_epoch()
+        self._crypto = self._core._crypto
+        app = flask.Flask(__name__)
+        bindPort = int(config.get('client.client.port', 59496))
+        self.bindPort = bindPort
+
+        self.clientToken = config.get('client.webpassword')
+        self.timeBypassToken = base64.b16encode(os.urandom(32)).decode()
+
+        self.publicAPI = None # gets set when the thread calls our setter... bad hack but kinda necessary with flask
+        #threading.Thread(target=PublicAPI, args=(self,)).start()
+        self.host = httpapi.apiutils.setbindip.set_bind_IP(self._core.privateApiHostFile, self._core)
+        logger.info('Running api on %s:%s' % (self.host, self.bindPort))
+        self.httpServer = ''
+
+        self.queueResponse = {}
+        self.get_block_data = httpapi.apiutils.GetBlockData(self)
+        onionrInst.setClientAPIInst(self)
+        register_private_blueprints.register_private_blueprints(self, app)
+        httpapi.load_plugin_blueprints(app)
+
+        self.httpServer = WSGIServer((self.host, bindPort), app, log=None, handler_class=httpapi.fdsafehandler.FDSafeHandler)
+        self.httpServer.serve_forever()
+
+    def setPublicAPIInstance(self, inst):
+        self.publicAPI = inst
+
+    def validateToken(self, token):
+        '''
+            Validate that the client token matches the given token. Used to prevent CSRF and data exfiltration
+        '''
+        if len(self.clientToken) == 0:
+            logger.error("client password needs to be set")
+            return False
+        try:
+            if not hmac.compare_digest(self.clientToken, token):
+                return False
+            else:
+                return True
+        except TypeError:
+            return False
+
+    def getUptime(self):
+        while True:
+            try:
+                return epoch.get_epoch() - self.startTime
+            except (AttributeError, NameError):
+                # Don't error on race condition with startup
+                pass
+
+    def getBlockData(self, bHash, decrypt=False, raw=False, headerOnly=False):
+        return self.get_block_data.get_block_data(bHash, decrypt=decrypt, raw=raw, headerOnly=headerOnly)

onionr/apiservers/private/register_private_blueprints.py

@@ -0,0 +1,33 @@
+'''
+    Onionr - Private P2P Communication
+
+    This file registers blueprints for the private api server
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import os
+from httpapi import security, friendsapi, profilesapi, configapi, insertblock, miscclientapi, onionrsitesapi, apiutils
+def register_private_blueprints(private_api, app):
+    app.register_blueprint(security.client.ClientAPISecurity(private_api).client_api_security_bp)
+    app.register_blueprint(friendsapi.friends)
+    app.register_blueprint(profilesapi.profile_BP)
+    app.register_blueprint(configapi.config_BP)
+    app.register_blueprint(insertblock.ib)
+    app.register_blueprint(miscclientapi.getblocks.client_get_blocks)
+    app.register_blueprint(miscclientapi.endpoints.PrivateEndpoints(private_api).private_endpoints_bp)
+    app.register_blueprint(onionrsitesapi.site_api)
+    app.register_blueprint(apiutils.shutdown.shutdown_bp)
+    app.register_blueprint(miscclientapi.staticfiles.static_files_bp)
+    return app

onionr/apiservers/public/__init__.py

@@ -0,0 +1,56 @@
+'''
+    Onionr - Private P2P Communication
+
+    This file handles all incoming http requests to the public api server, using Flask
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import time
+import flask
+from gevent.pywsgi import WSGIServer
+from httpapi import apiutils, security, fdsafehandler, miscpublicapi
+import logger, onionr
+class PublicAPI:
+    '''
+        The new client api server, isolated from the public api
+    '''
+    def __init__(self, clientAPI):
+        config = clientAPI.config
+        app = flask.Flask('PublicAPI')
+        app.config['MAX_CONTENT_LENGTH'] = 5 * 1024 * 1024
+        self.i2pEnabled = config.get('i2p.host', False)
+        self.hideBlocks = [] # Blocks to be denied sharing
+        self.host = apiutils.setbindip.set_bind_IP(clientAPI._core.publicApiHostFile, clientAPI._core)
+        self.torAdder = clientAPI._core.hsAddress
+        self.i2pAdder = clientAPI._core.i2pAddress
+        self.bindPort = config.get('client.public.port')
+        self.lastRequest = 0
+        self.hitCount = 0 # total rec requests to public api since server started
+        self.config = config
+        self.clientAPI = clientAPI
+        self.API_VERSION = onionr.API_VERSION
+        logger.info('Running public api on %s:%s' % (self.host, self.bindPort))
+
+        # Set instances, then startup our public api server
+        clientAPI.setPublicAPIInstance(self)
+        while self.torAdder == '':
+            clientAPI._core.refreshFirstStartVars()
+            self.torAdder = clientAPI._core.hsAddress
+            time.sleep(0.1)
+        
+        app.register_blueprint(security.public.PublicAPISecurity(self).public_api_security_bp)
+        app.register_blueprint(miscpublicapi.endpoints.PublicEndpoints(self).public_endpoints_bp)
+        self.httpServer = WSGIServer((self.host, self.bindPort), app, log=None, handler_class=fdsafehandler.FDSafeHandler)
+        self.httpServer.serve_forever()

onionr/blockimporter.py

@@ -1,5 +1,5 @@
 '''
-    Onionr - P2P Microblogging Platform & Social network
+    Onionr - Private P2P Communication
 
     Import block data and save it
 '''
@@ -18,6 +18,7 @@
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 import core, onionrexceptions, logger
+from onionrutils import validatemetadata, blockmetadata
 def importBlockFromData(content, coreInst):
     retData = False
 
@@ -34,17 +35,17 @@ def importBlockFromData(content, coreInst):
     except AttributeError:
         pass
 
-    metas = coreInst._utils.getBlockMetadataFromData(content) # returns tuple(metadata, meta), meta is also in metadata
+    metas = blockmetadata.get_block_metadata_from_data(content) # returns tuple(metadata, meta), meta is also in metadata
     metadata = metas[0]
-    if coreInst._utils.validateMetadata(metadata, metas[2]): # check if metadata is valid
+    if validatemetadata.validate_metadata(coreInst, metadata, metas[2]): # check if metadata is valid
         if coreInst._crypto.verifyPow(content): # check if POW is enough/correct
-            logger.info('Block passed proof, saving.')
+            logger.info('Block passed proof, saving.', terminal=True)
             try:
                 blockHash = coreInst.setData(content)
             except onionrexceptions.DiskAllocationReached:
                 pass
             else:
                 coreInst.addToBlockDB(blockHash, dataSaved=True)
-                coreInst._utils.processBlockMetadata(blockHash) # caches block metadata values to block database
+                blockmetadata.process_block_metadata(coreInst, blockHash) # caches block metadata values to block database
                 retData = True
     return retData

onionr/communicator.py

@@ -1,646 +0,0 @@
-#!/usr/bin/env python3
-'''
-    Onionr - P2P Anonymous Storage Network
-
-    This file contains both the OnionrCommunicate class for communcating with peers
-    and code to operate as a daemon, getting commands from the command queue database (see core.Core.daemonQueue)
-'''
-'''
-    This program is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation, either version 3 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License
-    along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
-import sys, os, core, config, json, requests, time, logger, threading, base64, onionr, uuid, binascii
-from dependencies import secrets
-from utils import networkmerger
-import onionrexceptions, onionrpeers, onionrevents as events, onionrplugins as plugins, onionrblockapi as block
-from communicatorutils import onionrdaemontools
-import onionrsockets, onionr, onionrproofs
-from communicatorutils import onionrcommunicatortimers, proxypicker
-
-OnionrCommunicatorTimers = onionrcommunicatortimers.OnionrCommunicatorTimers
-
-config.reload()
-class OnionrCommunicatorDaemon:
-    def __init__(self, onionrInst, proxyPort, developmentMode=config.get('general.dev_mode', False)):
-        onionrInst.communicatorInst = self
-        # configure logger and stuff
-        onionr.Onionr.setupConfig('data/', self = self)
-        self.proxyPort = proxyPort
-
-        self.isOnline = True # Assume we're connected to the internet
-
-        # list of timer instances
-        self.timers = []
-
-        # initialize core with Tor socks port being 3rd argument
-        self.proxyPort = proxyPort
-        self._core = onionrInst.onionrCore
-
-        self.blocksToUpload = []
-
-        # loop time.sleep delay in seconds
-        self.delay = 1
-
-        # lists of connected peers and peers we know we can't reach currently
-        self.onlinePeers = []
-        self.offlinePeers = []
-        self.cooldownPeer = {}
-        self.connectTimes = {}
-        self.peerProfiles = [] # list of peer's profiles (onionrpeers.PeerProfile instances)
-        self.newPeers = [] # Peers merged to us. Don't add to db until we know they're reachable
-
-        # amount of threads running by name, used to prevent too many
-        self.threadCounts = {}
-
-        # set true when shutdown command received
-        self.shutdown = False
-
-        # list of new blocks to download, added to when new block lists are fetched from peers
-        self.blockQueue = {}
-
-        # list of blocks currently downloading, avoid s
-        self.currentDownloading = []
-
-        # timestamp when the last online node was seen
-        self.lastNodeSeen = None
-
-        # Dict of time stamps for peer's block list lookup times, to avoid downloading full lists all the time
-        self.dbTimestamps = {}
-
-        # Clear the daemon queue for any dead messages
-        if os.path.exists(self._core.queueDB):
-            self._core.clearDaemonQueue()
-
-        # Loads in and starts the enabled plugins
-        plugins.reload()
-
-        # daemon tools are misc daemon functions, e.g. announce to online peers
-        # intended only for use by OnionrCommunicatorDaemon
-        self.daemonTools = onionrdaemontools.DaemonTools(self)
-
-        # time app started running for info/statistics purposes
-        self.startTime = self._core._utils.getEpoch()
-
-        if developmentMode:
-            OnionrCommunicatorTimers(self, self.heartbeat, 30)
-
-        # Set timers, function reference, seconds
-        # requiresPeer True means the timer function won't fire if we have no connected peers
-        peerPoolTimer = OnionrCommunicatorTimers(self, self.getOnlinePeers, 60, maxThreads=1)
-        OnionrCommunicatorTimers(self, self.runCheck, 2, maxThreads=1)
-        OnionrCommunicatorTimers(self, self.lookupBlocks, self._core.config.get('timers.lookupBlocks'), requiresPeer=True, maxThreads=1)
-        OnionrCommunicatorTimers(self, self.getBlocks, self._core.config.get('timers.getBlocks'), requiresPeer=True, maxThreads=2)
-        OnionrCommunicatorTimers(self, self.clearOfflinePeer, 58)
-        blockCleanupTimer = OnionrCommunicatorTimers(self, self.daemonTools.cleanOldBlocks, 65)
-        OnionrCommunicatorTimers(self, self.lookupAdders, 60, requiresPeer=True)
-        OnionrCommunicatorTimers(self, self.daemonTools.cooldownPeer, 30, requiresPeer=True)
-        OnionrCommunicatorTimers(self, self.uploadBlock, 10, requiresPeer=True, maxThreads=1)
-        OnionrCommunicatorTimers(self, self.daemonCommands, 6, maxThreads=1)
-        OnionrCommunicatorTimers(self, self.detectAPICrash, 30, maxThreads=1)
-        deniableBlockTimer = OnionrCommunicatorTimers(self, self.daemonTools.insertDeniableBlock, 180, requiresPeer=True, maxThreads=1)
-
-        netCheckTimer = OnionrCommunicatorTimers(self, self.daemonTools.netCheck, 600)
-        if config.get('general.security_level') == 0:
-            announceTimer = OnionrCommunicatorTimers(self, self.daemonTools.announceNode, 3600, requiresPeer=True, maxThreads=1)
-            announceTimer.count = (announceTimer.frequency - 120)
-        else:
-            logger.debug('Will not announce node.')
-        cleanupTimer = OnionrCommunicatorTimers(self, self.peerCleanup, 300, requiresPeer=True)
-        forwardSecrecyTimer = OnionrCommunicatorTimers(self, self.daemonTools.cleanKeys, 15, maxThreads=1)
-
-        # set loop to execute instantly to load up peer pool (replaced old pool init wait)
-        peerPoolTimer.count = (peerPoolTimer.frequency - 1)
-        cleanupTimer.count = (cleanupTimer.frequency - 60)
-        deniableBlockTimer.count = (deniableBlockTimer.frequency - 175)
-        blockCleanupTimer.count = (blockCleanupTimer.frequency - 5)
-        #forwardSecrecyTimer.count = (forwardSecrecyTimer.frequency - 990)
-
-        if config.get('general.socket_servers'):
-            self.socketServer = threading.Thread(target=onionrsockets.OnionrSocketServer, args=(self._core,))
-            self.socketServer.start()
-            self.socketClient = onionrsockets.OnionrSocketClient(self._core)
-
-        # Main daemon loop, mainly for calling timers, don't do any complex operations here to avoid locking
-        try:
-            while not self.shutdown:
-                for i in self.timers:
-                    if self.shutdown:
-                        break
-                    i.processTimer()
-                time.sleep(self.delay)
-                # Debug to print out used FDs (regular and net)
-                #proc = psutil.Process()
-                #print(proc.open_files(), len(psutil.net_connections()))
-        except KeyboardInterrupt:
-            self.shutdown = True
-            pass
-
-        logger.info('Goodbye.')
-        self._core.killSockets = True
-        self._core._utils.localCommand('shutdown') # shutdown the api
-        time.sleep(0.5)
-
-    def lookupAdders(self):
-        '''Lookup new peer addresses'''
-        logger.info('Looking up new addresses...')
-        tryAmount = 1
-        newPeers = []
-        for i in range(tryAmount):
-            # Download new peer address list from random online peers
-            if len(newPeers) > 10000:
-                # Dont get new peers if we have too many queued up
-                break
-            peer = self.pickOnlinePeer()
-            newAdders = self.peerAction(peer, action='pex')
-            try:
-                newPeers = newAdders.split(',')
-            except AttributeError:
-                pass
-        else:
-            # Validate new peers are good format and not already in queue
-            invalid = []
-            for x in newPeers:
-                x = x.strip()
-                if not self._core._utils.validateID(x) or x in self.newPeers or x == self._core.hsAddress:
-                    invalid.append(x)
-            for x in invalid:
-                newPeers.remove(x)
-            self.newPeers.extend(newPeers)
-        self.decrementThreadCount('lookupAdders')
-
-    def lookupBlocks(self):
-        '''Lookup new blocks & add them to download queue'''
-        logger.info('Looking up new blocks...')
-        tryAmount = 2
-        newBlocks = ''
-        existingBlocks = self._core.getBlockList()
-        triedPeers = [] # list of peers we've tried this time around
-        maxBacklog = 1560 # Max amount of *new* block hashes to have already in queue, to avoid memory exhaustion
-        lastLookupTime = 0 # Last time we looked up a particular peer's list
-        for i in range(tryAmount):
-            listLookupCommand = 'getblocklist' # This is defined here to reset it each time
-            if len(self.blockQueue) >= maxBacklog:
-                break
-            if not self.isOnline:
-                break
-            # check if disk allocation is used
-            if self._core._utils.storageCounter.isFull():
-                logger.debug('Not looking up new blocks due to maximum amount of allowed disk space used')
-                break
-            peer = self.pickOnlinePeer() # select random online peer
-            # if we've already tried all the online peers this time around, stop
-            if peer in triedPeers:
-                if len(self.onlinePeers) == len(triedPeers):
-                    break
-                else:
-                    continue
-            triedPeers.append(peer)
-
-            # Get the last time we looked up a peer's stamp to only fetch blocks since then.
-            # Saved in memory only for privacy reasons
-            try:
-                lastLookupTime = self.dbTimestamps[peer]
-            except KeyError:
-                lastLookupTime = 0
-            else:
-                listLookupCommand += '?date=%s' % (lastLookupTime,)
-            try:
-                newBlocks = self.peerAction(peer, listLookupCommand) # get list of new block hashes
-            except Exception as error:
-                logger.warn('Could not get new blocks from %s.' % peer, error = error)
-                newBlocks = False
-            else:
-                self.dbTimestamps[peer] = self._core._utils.getRoundedEpoch(roundS=60)
-            if newBlocks != False:
-                # if request was a success
-                for i in newBlocks.split('\n'):
-                    if self._core._utils.validateHash(i):
-                        # if newline seperated string is valid hash
-                        if not i in existingBlocks:
-                            # if block does not exist on disk and is not already in block queue
-                            if i not in self.blockQueue:
-                                if onionrproofs.hashMeetsDifficulty(i) and not self._core._blacklist.inBlacklist(i):
-                                    if len(self.blockQueue) <= 1000000:
-                                        self.blockQueue[i] = [peer] # add blocks to download queue
-                            else:
-                                if peer not in self.blockQueue[i]:
-                                    if len(self.blockQueue[i]) < 10:
-                                        self.blockQueue[i].append(peer)
-        self.decrementThreadCount('lookupBlocks')
-        return
-
-    def getBlocks(self):
-        '''download new blocks in queue'''
-        for blockHash in list(self.blockQueue):
-            triedQueuePeers = [] # List of peers we've tried for a block
-            try:
-                blockPeers = list(self.blockQueue[blockHash])
-            except KeyError:
-                blockPeers = []
-            removeFromQueue = True
-            if self.shutdown or not self.isOnline:
-                # Exit loop if shutting down or offline
-                break
-            # Do not download blocks being downloaded or that are already saved (edge cases)
-            if blockHash in self.currentDownloading:
-                #logger.debug('Already downloading block %s...' % blockHash)
-                continue
-            if blockHash in self._core.getBlockList():
-                #logger.debug('Block %s is already saved.' % (blockHash,))
-                try:
-                    del self.blockQueue[blockHash]
-                except KeyError:
-                    pass
-                continue
-            if self._core._blacklist.inBlacklist(blockHash):
-                continue
-            if self._core._utils.storageCounter.isFull():
-                break
-            self.currentDownloading.append(blockHash) # So we can avoid concurrent downloading in other threads of same block
-            if len(blockPeers) == 0:
-                peerUsed = self.pickOnlinePeer()
-            else:
-                blockPeers = self._core._crypto.randomShuffle(blockPeers)
-                peerUsed = blockPeers.pop(0)
-
-            if not self.shutdown and peerUsed.strip() != '':
-                logger.info("Attempting to download %s from %s..." % (blockHash[:12], peerUsed))
-            content = self.peerAction(peerUsed, 'getdata/' + blockHash) # block content from random peer (includes metadata)
-            if content != False and len(content) > 0:
-                try:
-                    content = content.encode()
-                except AttributeError:
-                    pass
-
-                realHash = self._core._crypto.sha3Hash(content)
-                try:
-                    realHash = realHash.decode() # bytes on some versions for some reason
-                except AttributeError:
-                    pass
-                if realHash == blockHash:
-                    content = content.decode() # decode here because sha3Hash needs bytes above
-                    metas = self._core._utils.getBlockMetadataFromData(content) # returns tuple(metadata, meta), meta is also in metadata
-                    metadata = metas[0]
-                    if self._core._utils.validateMetadata(metadata, metas[2]): # check if metadata is valid, and verify nonce
-                        if self._core._crypto.verifyPow(content): # check if POW is enough/correct
-                            logger.info('Attempting to save block %s...' % blockHash[:12])
-                            try:
-                                self._core.setData(content)
-                            except onionrexceptions.DiskAllocationReached:
-                                logger.error('Reached disk allocation allowance, cannot save block %s.' % blockHash)
-                                removeFromQueue = False
-                            else:
-                                self._core.addToBlockDB(blockHash, dataSaved=True)
-                                self._core._utils.processBlockMetadata(blockHash) # caches block metadata values to block database
-                        else:
-                            logger.warn('POW failed for block %s.' % blockHash)
-                    else:
-                        if self._core._blacklist.inBlacklist(realHash):
-                            logger.warn('Block %s is blacklisted.' % (realHash,))
-                        else:
-                            logger.warn('Metadata for block %s is invalid.' % blockHash)
-                            self._core._blacklist.addToDB(blockHash)
-                else:
-                    # if block didn't meet expected hash
-                    tempHash = self._core._crypto.sha3Hash(content) # lazy hack, TODO use var
-                    try:
-                        tempHash = tempHash.decode()
-                    except AttributeError:
-                        pass
-                    # Punish peer for sharing invalid block (not always malicious, but is bad regardless)
-                    onionrpeers.PeerProfiles(peerUsed, self._core).addScore(-50)
-                    if tempHash != 'ed55e34cb828232d6c14da0479709bfa10a0923dca2b380496e6b2ed4f7a0253':
-                        # Dumb hack for 404 response from peer. Don't log it if 404 since its likely not malicious or a critical error.
-                        logger.warn('Block hash validation failed for ' + blockHash + ' got ' + tempHash)
-                    else:
-                        removeFromQueue = False # Don't remove from queue if 404
-                if removeFromQueue:
-                    try:
-                        del self.blockQueue[blockHash] # remove from block queue both if success or false
-                    except KeyError:
-                        pass
-            self.currentDownloading.remove(blockHash)
-        self.decrementThreadCount('getBlocks')
-        return
-
-    def pickOnlinePeer(self):
-        '''randomly picks peer from pool without bias (using secrets module)'''
-        retData = ''
-        while True:
-            peerLength = len(self.onlinePeers)
-            if peerLength <= 0:
-                break
-            try:
-                # get a random online peer, securely. May get stuck in loop if network is lost or if all peers in pool magically disconnect at once
-                retData = self.onlinePeers[self._core._crypto.secrets.randbelow(peerLength)]
-            except IndexError:
-                pass
-            else:
-                break
-        return retData
-
-    def decrementThreadCount(self, threadName):
-        '''Decrement amount of a thread name if more than zero, called when a function meant to be run in a thread ends'''
-        try:
-            if self.threadCounts[threadName] > 0:
-                self.threadCounts[threadName] -= 1
-        except KeyError:
-            pass
-
-    def clearOfflinePeer(self):
-        '''Removes the longest offline peer to retry later'''
-        try:
-            removed = self.offlinePeers.pop(0)
-        except IndexError:
-            pass
-        else:
-            logger.debug('Removed ' + removed + ' from offline list, will try them again.')
-        self.decrementThreadCount('clearOfflinePeer')
-
-    def getOnlinePeers(self):
-        '''
-            Manages the self.onlinePeers attribute list, connects to more peers if we have none connected
-        '''
-
-        logger.debug('Refreshing peer pool...')
-        maxPeers = int(config.get('peers.max_connect', 10))
-        needed = maxPeers - len(self.onlinePeers)
-
-        for i in range(needed):
-            if len(self.onlinePeers) == 0:
-                self.connectNewPeer(useBootstrap=True)
-            else:
-                self.connectNewPeer()
-
-            if self.shutdown:
-                break
-        else:
-            if len(self.onlinePeers) == 0:
-                logger.debug('Couldn\'t connect to any peers.' + (' Last node seen %s ago.' % self.daemonTools.humanReadableTime(time.time() - self.lastNodeSeen) if not self.lastNodeSeen is None else ''))
-            else:
-                self.lastNodeSeen = time.time()
-        self.decrementThreadCount('getOnlinePeers')
-
-    def addBootstrapListToPeerList(self, peerList):
-        '''
-            Add the bootstrap list to the peer list (no duplicates)
-        '''
-        for i in self._core.bootstrapList:
-            if i not in peerList and i not in self.offlinePeers and i != self._core.hsAddress and len(str(i).strip()) > 0:
-                peerList.append(i)
-                self._core.addAddress(i)
-
-    def connectNewPeer(self, peer='', useBootstrap=False):
-        '''Adds a new random online peer to self.onlinePeers'''
-        retData = False
-        tried = self.offlinePeers
-        if peer != '':
-            if self._core._utils.validateID(peer):
-                peerList = [peer]
-            else:
-                raise onionrexceptions.InvalidAddress('Will not attempt connection test to invalid address')
-        else:
-            peerList = self._core.listAdders()
-
-        mainPeerList = self._core.listAdders()
-        peerList = onionrpeers.getScoreSortedPeerList(self._core)
-
-        if len(peerList) < 8 or secrets.randbelow(4) == 3:
-            tryingNew = []
-            for x in self.newPeers:
-                if x not in peerList:
-                    peerList.append(x)
-                    tryingNew.append(x)
-            for i in tryingNew:
-                self.newPeers.remove(i)
-        
-        if len(peerList) == 0 or useBootstrap:
-            # Avoid duplicating bootstrap addresses in peerList
-            self.addBootstrapListToPeerList(peerList)
-
-        for address in peerList:
-            if not config.get('tor.v3onions') and len(address) == 62:
-                continue
-            if address == self._core.hsAddress:
-                continue
-            if len(address) == 0 or address in tried or address in self.onlinePeers or address in self.cooldownPeer:
-                continue
-            if self.shutdown:
-                return
-            if self.peerAction(address, 'ping') == 'pong!':
-                time.sleep(0.1)
-                if address not in mainPeerList:
-                    networkmerger.mergeAdders(address, self._core)
-                if address not in self.onlinePeers:
-                    logger.info('Connected to ' + address)
-                    self.onlinePeers.append(address)
-                    self.connectTimes[address] = self._core._utils.getEpoch()
-                retData = address
-
-                # add peer to profile list if they're not in it
-                for profile in self.peerProfiles:
-                    if profile.address == address:
-                        break
-                else:
-                    self.peerProfiles.append(onionrpeers.PeerProfiles(address, self._core))
-                break
-            else:
-                tried.append(address)
-                logger.debug('Failed to connect to ' + address)
-        return retData
-
-    def removeOnlinePeer(self, peer):
-        '''Remove an online peer'''
-        try:
-            del self.connectTimes[peer]
-        except KeyError:
-            pass
-        try:
-            del self.dbTimestamps[peer]
-        except KeyError:
-            pass
-        try:
-            self.onlinePeers.remove(peer)
-        except ValueError:
-            pass
-
-    def peerCleanup(self):
-        '''This just calls onionrpeers.cleanupPeers, which removes dead or bad peers (offline too long, too slow)'''
-        onionrpeers.peerCleanup(self._core)
-        self.decrementThreadCount('peerCleanup')
-
-    def printOnlinePeers(self):
-        '''logs online peer list'''
-        if len(self.onlinePeers) == 0:
-            logger.warn('No online peers')
-        else:
-            logger.info('Online peers:')
-            for i in self.onlinePeers:
-                score = str(self.getPeerProfileInstance(i).score)
-                logger.info(i + ', score: ' + score)
-
-    def peerAction(self, peer, action, data='', returnHeaders=False):
-        '''Perform a get request to a peer'''
-        if len(peer) == 0:
-            return False
-        #logger.debug('Performing ' + action + ' with ' + peer + ' on port ' + str(self.proxyPort))
-        url = 'http://%s/%s' % (peer, action)
-        if len(data) > 0:
-            url += '&data=' + data
-
-        self._core.setAddressInfo(peer, 'lastConnectAttempt', self._core._utils.getEpoch()) # mark the time we're trying to request this peer
-
-        retData = self._core._utils.doGetRequest(url, port=self.proxyPort)
-        # if request failed, (error), mark peer offline
-        if retData == False:
-            try:
-                self.getPeerProfileInstance(peer).addScore(-10)
-                self.removeOnlinePeer(peer)
-                if action != 'ping':
-                    self.getOnlinePeers() # Will only add a new peer to pool if needed
-            except ValueError:
-                pass
-        else:
-            self._core.setAddressInfo(peer, 'lastConnect', self._core._utils.getEpoch())
-            self.getPeerProfileInstance(peer).addScore(1)
-        return retData # If returnHeaders, returns tuple of data, headers. if not, just data string
-
-    def getPeerProfileInstance(self, peer):
-        '''Gets a peer profile instance from the list of profiles, by address name'''
-        for i in self.peerProfiles:
-            # if the peer's profile is already loaded, return that
-            if i.address == peer:
-                retData = i
-                break
-        else:
-            # if the peer's profile is not loaded, return a new one. connectNewPeer adds it the list on connect
-            retData = onionrpeers.PeerProfiles(peer, self._core)
-        return retData
-
-    def getUptime(self):
-        return self._core._utils.getEpoch() - self.startTime
-
-    def heartbeat(self):
-        '''Show a heartbeat debug message'''
-        logger.debug('Heartbeat. Node running for %s.' % self.daemonTools.humanReadableTime(self.getUptime()))
-        self.decrementThreadCount('heartbeat')
-
-    def daemonCommands(self):
-        '''
-            Process daemon commands from daemonQueue
-        '''
-        cmd = self._core.daemonQueue()
-        response = ''
-        if cmd is not False:
-            events.event('daemon_command', onionr = None, data = {'cmd' : cmd})
-            if cmd[0] == 'shutdown':
-                self.shutdown = True
-            elif cmd[0] == 'announceNode':
-                if len(self.onlinePeers) > 0:
-                    self.announce(cmd[1])
-                else:
-                    logger.debug("No nodes connected. Will not introduce node.")
-            elif cmd[0] == 'runCheck': # deprecated
-                logger.debug('Status check; looks good.')
-                open(self._core.dataDir + '.runcheck', 'w+').close()
-            elif cmd[0] == 'connectedPeers':
-                response = '\n'.join(list(self.onlinePeers)).strip()
-                if response == '':
-                    response = 'none'
-            elif cmd[0] == 'localCommand':
-                response = self._core._utils.localCommand(cmd[1])
-            elif cmd[0] == 'pex':
-                for i in self.timers:
-                    if i.timerFunction.__name__ == 'lookupAdders':
-                        i.count = (i.frequency - 1)
-            elif cmd[0] == 'uploadBlock':
-                self.blocksToUpload.append(cmd[1])
-            elif cmd[0] == 'startSocket':
-                # Create our own socket server
-                socketInfo = json.loads(cmd[1])
-                socketInfo['id'] = uuid.uuid4()
-                self._core.startSocket = socketInfo
-            elif cmd[0] == 'addSocket':
-                # Socket server was created for us
-                socketInfo = json.loads(cmd[1])
-                peer = socketInfo['peer']
-                reason = socketInfo['reason']
-                threading.Thread(target=self.socketClient.startSocket, args=(peer, reason)).start()
-            else:
-                logger.info('Recieved daemonQueue command:' + cmd[0])
-
-            if cmd[0] not in ('', None):
-                if response != '':
-                    self._core._utils.localCommand('queueResponseAdd/' + cmd[4], post=True, postData={'data': response})
-            response = ''
-
-        self.decrementThreadCount('daemonCommands')
-
-    def uploadBlock(self):
-        '''Upload our block to a few peers'''
-        # when inserting a block, we try to upload it to a few peers to add some deniability
-        triedPeers = []
-        finishedUploads = []
-        self.blocksToUpload = self._core._crypto.randomShuffle(self.blocksToUpload)
-        if len(self.blocksToUpload) != 0:
-            for bl in self.blocksToUpload:
-                if not self._core._utils.validateHash(bl):
-                    logger.warn('Requested to upload invalid block')
-                    self.decrementThreadCount('uploadBlock')
-                    return
-                for i in range(min(len(self.onlinePeers), 6)):
-                    peer = self.pickOnlinePeer()
-                    if peer in triedPeers:
-                        continue
-                    triedPeers.append(peer)
-                    url = 'http://' + peer + '/upload'
-                    data = {'block': block.Block(bl).getRaw()}
-                    proxyType = proxypicker.pick_proxy(peer)
-                    logger.info("Uploading block to " + peer)
-                    if not self._core._utils.doPostRequest(url, data=data, proxyType=proxyType) == False:
-                        self._core._utils.localCommand('waitforshare/' + bl, post=True)
-                        finishedUploads.append(bl)
-        for x in finishedUploads:
-            try:
-                self.blocksToUpload.remove(x)
-            except ValueError:
-                pass
-        self.decrementThreadCount('uploadBlock')
-
-    def announce(self, peer):
-        '''Announce to peers our address'''
-        if self.daemonTools.announceNode() == False:
-            logger.warn('Could not introduce node.')
-
-    def detectAPICrash(self):
-        '''exit if the api server crashes/stops'''
-        if self._core._utils.localCommand('ping', silent=False) not in ('pong', 'pong!'):
-            for i in range(8):
-                if self._core._utils.localCommand('ping') in ('pong', 'pong!') or self.shutdown:
-                    break # break for loop
-                time.sleep(1)
-            else:
-                # This executes if the api is NOT detected to be running
-                events.event('daemon_crash', onionr = None, data = {})
-                logger.error('Daemon detected API crash (or otherwise unable to reach API after long time), stopping...')
-                self.shutdown = True
-        self.decrementThreadCount('detectAPICrash')
-
-    def runCheck(self):
-        if self.daemonTools.runCheck():
-            logger.debug('Status check; looks good.')
-
-        self.decrementThreadCount('runCheck')
-
-def startCommunicator(onionrInst, proxyPort):
-    OnionrCommunicatorDaemon(onionrInst, proxyPort)

onionr/communicator/__init__.py

@@ -0,0 +1,254 @@
+'''
+    Onionr - Private P2P Communication
+
+    This file contains both the OnionrCommunicate class for communcating with peers
+    and code to operate as a daemon, getting commands from the command queue database (see core.Core.daemonQueue)
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sys, os, time
+import core, config, logger, onionr
+import onionrexceptions, onionrpeers, onionrevents as events, onionrplugins as plugins, onionrblockapi as block
+from . import onlinepeers
+from communicatorutils import servicecreator, onionrcommunicatortimers
+from communicatorutils import downloadblocks, lookupblocks, lookupadders
+from communicatorutils import servicecreator, connectnewpeers, uploadblocks
+from communicatorutils import daemonqueuehandler, announcenode, deniableinserts
+from communicatorutils import cooldownpeer, housekeeping, netcheck
+from onionrutils import localcommand, epoch
+from etc import humanreadabletime
+import onionrservices, onionr, onionrproofs
+OnionrCommunicatorTimers = onionrcommunicatortimers.OnionrCommunicatorTimers
+
+config.reload()
+class OnionrCommunicatorDaemon:
+    def __init__(self, onionrInst, proxyPort, developmentMode=config.get('general.dev_mode', False)):
+        onionrInst.communicatorInst = self
+        # configure logger and stuff
+        onionr.Onionr.setupConfig('data/', self = self)
+        self.proxyPort = proxyPort
+
+        self.isOnline = True # Assume we're connected to the internet
+
+        # list of timer instances
+        self.timers = []
+
+        # initialize core with Tor socks port being 3rd argument
+        self.proxyPort = proxyPort
+        self._core = onionrInst.onionrCore
+
+        self.blocksToUpload = []
+
+        # loop time.sleep delay in seconds
+        self.delay = 1
+
+        # lists of connected peers and peers we know we can't reach currently
+        self.onlinePeers = []
+        self.offlinePeers = []
+        self.cooldownPeer = {}
+        self.connectTimes = {}
+        self.peerProfiles = [] # list of peer's profiles (onionrpeers.PeerProfile instances)
+        self.newPeers = [] # Peers merged to us. Don't add to db until we know they're reachable
+        self.announceProgress = {}
+        self.announceCache = {}
+
+        # amount of threads running by name, used to prevent too many
+        self.threadCounts = {}
+
+        # set true when shutdown command received
+        self.shutdown = False
+
+        # list of new blocks to download, added to when new block lists are fetched from peers
+        self.blockQueue = {}
+
+        # list of blocks currently downloading, avoid s
+        self.currentDownloading = []
+
+        # timestamp when the last online node was seen
+        self.lastNodeSeen = None
+
+        # Dict of time stamps for peer's block list lookup times, to avoid downloading full lists all the time
+        self.dbTimestamps = {}
+
+        # Clear the daemon queue for any dead messages
+        if os.path.exists(self._core.queueDB):
+            self._core.clearDaemonQueue()
+
+        # Loads in and starts the enabled plugins
+        plugins.reload()
+
+        # time app started running for info/statistics purposes
+        self.startTime = epoch.get_epoch()
+
+        if developmentMode:
+            OnionrCommunicatorTimers(self, self.heartbeat, 30)
+
+        # Set timers, function reference, seconds
+        # requiresPeer True means the timer function won't fire if we have no connected peers
+        peerPoolTimer = OnionrCommunicatorTimers(self, onlinepeers.get_online_peers, 60, maxThreads=1, myArgs=[self])
+        OnionrCommunicatorTimers(self, self.runCheck, 2, maxThreads=1)
+
+        # Timers to periodically lookup new blocks and download them
+        OnionrCommunicatorTimers(self, self.lookupBlocks, self._core.config.get('timers.lookupBlocks', 25), requiresPeer=True, maxThreads=1)
+        OnionrCommunicatorTimers(self, self.getBlocks, self._core.config.get('timers.getBlocks', 30), requiresPeer=True, maxThreads=2)
+
+        # Timer to reset the longest offline peer so contact can be attempted again
+        OnionrCommunicatorTimers(self, onlinepeers.clear_offline_peer, 58, myArgs=[self])
+
+        # Timer to cleanup old blocks
+        blockCleanupTimer = OnionrCommunicatorTimers(self, housekeeping.clean_old_blocks, 65, myArgs=[self])
+
+        # Timer to discover new peers
+        OnionrCommunicatorTimers(self, self.lookupAdders, 60, requiresPeer=True)
+
+        # Timer for adjusting which peers we actively communicate to at any given time, to avoid over-using peers
+        OnionrCommunicatorTimers(self, cooldownpeer.cooldown_peer, 30, myArgs=[self], requiresPeer=True)
+
+        # Timer to read the upload queue and upload the entries to peers
+        OnionrCommunicatorTimers(self, uploadblocks.upload_blocks_from_communicator, 5, myArgs=[self], requiresPeer=True, maxThreads=1)
+
+        # Timer to process the daemon command queue
+        OnionrCommunicatorTimers(self, daemonqueuehandler.handle_daemon_commands, 6, myArgs=[self], maxThreads=3)
+
+        # Setup direct connections
+        if config.get('general.socket_servers', False):
+            self.services = onionrservices.OnionrServices(self._core)
+            self.active_services = []
+            self.service_greenlets = []
+            OnionrCommunicatorTimers(self, servicecreator.service_creator, 5, maxThreads=50, myArgs=[self])
+        else:
+            self.services = None
+        
+        # This timer creates deniable blocks, in an attempt to further obfuscate block insertion metadata
+        if config.get('general.insert_deniable_blocks', True):
+            deniableBlockTimer = OnionrCommunicatorTimers(self, deniableinserts.insert_deniable_block, 180, myArgs=[self], requiresPeer=True, maxThreads=1)
+            deniableBlockTimer.count = (deniableBlockTimer.frequency - 175)
+
+        # Timer to check for connectivity, through Tor to various high-profile onion services
+        netCheckTimer = OnionrCommunicatorTimers(self, netcheck.net_check, 600, myArgs=[self])
+
+        # Announce the public API server transport address to other nodes if security level allows
+        if config.get('general.security_level', 1) == 0 and config.get('general.announce_node', True):
+            # Default to high security level incase config breaks
+            announceTimer = OnionrCommunicatorTimers(self, announcenode.announce_node, 3600, myArgs=[self], requiresPeer=True, maxThreads=1)
+            announceTimer.count = (announceTimer.frequency - 120)
+        else:
+            logger.debug('Will not announce node.')
+        
+        # Timer to delete malfunctioning or long-dead peers
+        cleanupTimer = OnionrCommunicatorTimers(self, self.peerCleanup, 300, requiresPeer=True)
+
+        # Timer to cleanup dead ephemeral forward secrecy keys 
+        forwardSecrecyTimer = OnionrCommunicatorTimers(self, housekeeping.clean_keys, 15, myArgs=[self], maxThreads=1)
+
+        # Adjust initial timer triggers
+        peerPoolTimer.count = (peerPoolTimer.frequency - 1)
+        cleanupTimer.count = (cleanupTimer.frequency - 60)
+        blockCleanupTimer.count = (blockCleanupTimer.frequency - 5)
+
+        # Main daemon loop, mainly for calling timers, don't do any complex operations here to avoid locking
+        try:
+            while not self.shutdown:
+                for i in self.timers:
+                    if self.shutdown:
+                        break
+                    i.processTimer()
+                time.sleep(self.delay)
+                # Debug to print out used FDs (regular and net)
+                #proc = psutil.Process()
+                #print(proc.open_files(), len(psutil.net_connections()))
+        except KeyboardInterrupt:
+            self.shutdown = True
+            pass
+
+        logger.info('Goodbye. (Onionr is cleaning up, and will exit)', terminal=True)
+        try:
+            self.service_greenlets
+        except AttributeError:
+            pass
+        else:
+            for server in self.service_greenlets:
+                server.stop()
+        localcommand.local_command(self._core, 'shutdown') # shutdown the api
+        time.sleep(0.5)
+
+    def lookupAdders(self):
+        '''Lookup new peer addresses'''
+        lookupadders.lookup_new_peer_transports_with_communicator(self)
+
+    def lookupBlocks(self):
+        '''Lookup new blocks & add them to download queue'''
+        lookupblocks.lookup_blocks_from_communicator(self)
+
+    def getBlocks(self):
+        '''download new blocks in queue'''
+        downloadblocks.download_blocks_from_communicator(self)
+
+    def decrementThreadCount(self, threadName):
+        '''Decrement amount of a thread name if more than zero, called when a function meant to be run in a thread ends'''
+        try:
+            if self.threadCounts[threadName] > 0:
+                self.threadCounts[threadName] -= 1
+        except KeyError:
+            pass
+
+    def connectNewPeer(self, peer='', useBootstrap=False):
+        '''Adds a new random online peer to self.onlinePeers'''
+        connectnewpeers.connect_new_peer_to_communicator(self, peer, useBootstrap)
+
+    def peerCleanup(self):
+        '''This just calls onionrpeers.cleanupPeers, which removes dead or bad peers (offline too long, too slow)'''
+        onionrpeers.peer_cleanup(self._core)
+        self.decrementThreadCount('peerCleanup')
+
+    def getPeerProfileInstance(self, peer):
+        '''Gets a peer profile instance from the list of profiles, by address name'''
+        for i in self.peerProfiles:
+            # if the peer's profile is already loaded, return that
+            if i.address == peer:
+                retData = i
+                break
+        else:
+            # if the peer's profile is not loaded, return a new one. connectNewPeer adds it the list on connect
+            retData = onionrpeers.PeerProfiles(peer, self._core)
+        return retData
+
+    def getUptime(self):
+        return epoch.get_epoch() - self.startTime
+
+    def heartbeat(self):
+        '''Show a heartbeat debug message'''
+        logger.debug('Heartbeat. Node running for %s.' % humanreadabletime.human_readable_time(self.getUptime()))
+        self.decrementThreadCount('heartbeat')
+
+    def announce(self, peer):
+        '''Announce to peers our address'''
+        if announcenode.announce_node(self) == False:
+            logger.warn('Could not introduce node.', terminal=True)
+
+    def runCheck(self):
+        if run_file_exists(self):
+            logger.debug('Status check; looks good.')
+
+        self.decrementThreadCount('runCheck')
+
+def startCommunicator(onionrInst, proxyPort):
+    OnionrCommunicatorDaemon(onionrInst, proxyPort)
+
+def run_file_exists(daemon):
+    if os.path.isfile(daemon._core.dataDir + '.runcheck'):
+        os.remove(daemon._core.dataDir + '.runcheck')
+        return True
+    return False

onionr/communicator/bootstrappeers.py

@@ -0,0 +1,27 @@
+'''
+    Onionr - Private P2P Communication
+
+    add bootstrap peers to the communicator peer list
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+def add_bootstrap_list_to_peer_list(comm_inst, peerList):
+    '''
+        Add the bootstrap list to the peer list (no duplicates)
+    '''
+    for i in comm_inst._core.bootstrapList:
+        if i not in peerList and i not in comm_inst.offlinePeers and i != comm_inst._core.hsAddress and len(str(i).strip()) > 0:
+            peerList.append(i)
+            comm_inst._core.addAddress(i)

onionr/communicator/onlinepeers/__init__.py

@@ -0,0 +1,6 @@
+from . import clearofflinepeer, onlinepeers, pickonlinepeers, removeonlinepeer
+
+clear_offline_peer = clearofflinepeer.clear_offline_peer
+get_online_peers = onlinepeers.get_online_peers
+pick_online_peer = pickonlinepeers.pick_online_peer
+remove_online_peer = removeonlinepeer.remove_online_peer

onionr/communicator/onlinepeers/clearofflinepeer.py

@@ -0,0 +1,29 @@
+'''
+    Onionr - Private P2P Communication
+
+    clear offline peer in a communicator instance
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import logger
+def clear_offline_peer(comm_inst):
+    '''Removes the longest offline peer to retry later'''
+    try:
+        removed = comm_inst.offlinePeers.pop(0)
+    except IndexError:
+        pass
+    else:
+        logger.debug('Removed ' + removed + ' from offline list, will try them again.')
+    comm_inst.decrementThreadCount('clear_offline_peer')

onionr/communicator/onlinepeers/onlinepeers.py

@@ -0,0 +1,45 @@
+'''
+    Onionr - Private P2P Communication
+
+    get online peers in a communicator instance
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import time
+from etc import humanreadabletime
+import logger
+def get_online_peers(comm_inst):
+    '''
+        Manages the comm_inst.onlinePeers attribute list, connects to more peers if we have none connected
+    '''
+    config = comm_inst._core.config
+    logger.debug('Refreshing peer pool...')
+    maxPeers = int(config.get('peers.max_connect', 10))
+    needed = maxPeers - len(comm_inst.onlinePeers)
+
+    for i in range(needed):
+        if len(comm_inst.onlinePeers) == 0:
+            comm_inst.connectNewPeer(useBootstrap=True)
+        else:
+            comm_inst.connectNewPeer()
+
+        if comm_inst.shutdown:
+            break
+    else:
+        if len(comm_inst.onlinePeers) == 0:
+            logger.debug('Couldn\'t connect to any peers.' + (' Last node seen %s ago.' % humanreadabletime.human_readable_time(time.time() - comm_inst.lastNodeSeen) if not comm_inst.lastNodeSeen is None else ''), terminal=True)
+        else:
+            comm_inst.lastNodeSeen = time.time()
+    comm_inst.decrementThreadCount('get_online_peers')

onionr/communicator/onlinepeers/pickonlinepeers.py

@@ -0,0 +1,34 @@
+'''
+    Onionr - Private P2P Communication
+
+    pick online peers in a communicator instance
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+def pick_online_peer(comm_inst):
+    '''randomly picks peer from pool without bias (using secrets module)'''
+    retData = ''
+    while True:
+        peerLength = len(comm_inst.onlinePeers)
+        if peerLength <= 0:
+            break
+        try:
+            # get a random online peer, securely. May get stuck in loop if network is lost or if all peers in pool magically disconnect at once
+            retData = comm_inst.onlinePeers[comm_inst._core._crypto.secrets.randbelow(peerLength)]
+        except IndexError:
+            pass
+        else:
+            break
+    return retData

onionr/communicator/onlinepeers/removeonlinepeer.py

@@ -0,0 +1,33 @@
+'''
+    Onionr - Private P2P Communication
+
+    remove an online peer from the pool in a communicator instance
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+def remove_online_peer(comm_inst, peer):
+    '''Remove an online peer'''
+    try:
+        del comm_inst.connectTimes[peer]
+    except KeyError:
+        pass
+    try:
+        del comm_inst.dbTimestamps[peer]
+    except KeyError:
+        pass
+    try:
+        comm_inst.onlinePeers.remove(peer)
+    except ValueError:
+        pass

onionr/communicator/peeraction.py

@@ -0,0 +1,53 @@
+'''
+    Onionr - Private P2P Communication
+
+    This file implements logic for performing requests to Onionr peers
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import streamedrequests
+import logger
+from onionrutils import epoch, basicrequests
+from . import onlinepeers
+def peer_action(comm_inst, peer, action, data='', returnHeaders=False, max_resp_size=5242880):
+    '''Perform a get request to a peer'''
+    penalty_score = -10
+    if len(peer) == 0:
+        return False
+    url = 'http://%s/%s' % (peer, action)
+    if len(data) > 0:
+        url += '&data=' + data
+
+    comm_inst._core.setAddressInfo(peer, 'lastConnectAttempt', epoch.get_epoch()) # mark the time we're trying to request this peer
+    try:
+        retData = basicrequests.do_get_request(comm_inst._core, url, port=comm_inst.proxyPort, max_size=max_resp_size)
+    except streamedrequests.exceptions.ResponseLimitReached:
+        logger.warn('Request failed due to max response size being overflowed', terminal=True)
+        retData = False
+        penalty_score = -100
+    # if request failed, (error), mark peer offline
+    if retData == False:
+        try:
+            comm_inst.getPeerProfileInstance(peer).addScore(penalty_score)
+            onlinepeers.remove_online_peer(comm_inst, peer)
+            if action != 'ping' and not comm_inst.shutdown:
+                logger.warn('Lost connection to ' + peer, terminal=True)
+                onlinepeers.get_online_peers(comm_inst) # Will only add a new peer to pool if needed
+        except ValueError:
+            pass
+    else:
+        comm_inst._core.setAddressInfo(peer, 'lastConnect', epoch.get_epoch())
+        comm_inst.getPeerProfileInstance(peer).addScore(1)
+    return retData # If returnHeaders, returns tuple of data, headers. if not, just data string

onionr/communicatorutils/README.md

@@ -0,0 +1,33 @@
+# communicatorutils
+
+The files in this submodule handle various subtasks and utilities for the onionr communicator.
+
+## Files:
+
+announcenode.py: Uses a communicator instance to announce our transport address to connected nodes
+
+connectnewpeers.py: takes a communicator instance and has it connect to as many peers as needed, and/or to a new specified peer.
+
+cooldownpeer.py: randomly selects a connected peer in a communicator and disconnects them for the purpose of security and network balancing.
+
+daemonqueuehandler.py: checks for new commands in the daemon queue and processes them accordingly.
+
+deniableinserts.py: insert fake blocks with the communicator for plausible deniability
+
+downloadblocks.py: iterates a communicator instance's block download queue and attempts to download the blocks from online peers
+
+housekeeping.py: cleans old blocks and forward secrecy keys
+
+lookupadders.py: ask connected peers to share their list of peer transport addresses
+
+lookupblocks.py: lookup new blocks from connected peers from the communicator
+
+netcheck.py: check if the node is online based on communicator status and onion server ping results
+
+onionrcommunicataortimers.py: create a timer for a function to be launched on an interval. Control how many possible instances of a timer may be running a function at once and control if the timer should be ran in a thread or not.
+
+proxypicker.py: returns a string name for the appropriate proxy to be used with a particular peer transport address.
+
+servicecreator.py: iterate connection blocks and create new direct connection servers for them.
+
+uploadblocks.py: iterate a communicator's upload queue and upload the blocks to connected peers

onionr/communicatorutils/announcenode.py

@@ -0,0 +1,86 @@
+'''
+    Onionr - Private P2P Communication
+
+    Use a communicator instance to announce our transport address to connected nodes
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import base64
+import onionrproofs, logger
+from etc import onionrvalues
+from onionrutils import basicrequests, bytesconverter
+from communicator import onlinepeers
+
+def announce_node(daemon):
+    '''Announce our node to our peers'''
+    ov = onionrvalues.OnionrValues()
+    retData = False
+    announceFail = False
+    
+    # Do not let announceCache get too large
+    if len(daemon.announceCache) >= 10000:
+        daemon.announceCache.popitem()
+
+    if daemon._core.config.get('general.security_level', 0) == 0:
+        # Announce to random online peers
+        for i in daemon.onlinePeers:
+            if not i in daemon.announceCache and not i in daemon.announceProgress:
+                peer = i
+                break
+        else:
+            peer = onlinepeers.pick_online_peer(daemon)
+
+        for x in range(1):
+            if x == 1 and daemon._core.config.get('i2p.host'):
+                ourID = daemon._core.config.get('i2p.own_addr').strip()
+            else:
+                ourID = daemon._core.hsAddress.strip()
+
+            url = 'http://' + peer + '/announce'
+            data = {'node': ourID}
+
+            combinedNodes = ourID + peer
+            if ourID != 1:
+                #TODO: Extend existingRand for i2p
+                existingRand = bytesconverter.bytes_to_str(daemon._core.getAddressInfo(peer, 'powValue'))
+                # Reset existingRand if it no longer meets the minimum POW
+                if type(existingRand) is type(None) or not existingRand.endswith('0' * ov.announce_pow):
+                    existingRand = ''
+
+            if peer in daemon.announceCache:
+                data['random'] = daemon.announceCache[peer]
+            elif len(existingRand) > 0:
+                data['random'] = existingRand
+            else:
+                daemon.announceProgress[peer] = True
+                proof = onionrproofs.DataPOW(combinedNodes, forceDifficulty=ov.announce_pow)
+                del daemon.announceProgress[peer]
+                try:
+                    data['random'] = base64.b64encode(proof.waitForResult()[1])
+                except TypeError:
+                    # Happens when we failed to produce a proof
+                    logger.error("Failed to produce a pow for announcing to " + peer)
+                    announceFail = True
+                else:
+                    daemon.announceCache[peer] = data['random']
+            if not announceFail:
+                logger.info('Announcing node to ' + url)
+                if basicrequests.do_post_request(daemon._core, url, data) == 'Success':
+                    logger.info('Successfully introduced node to ' + peer, terminal=True)
+                    retData = True
+                    daemon._core.setAddressInfo(peer, 'introduced', 1)
+                    daemon._core.setAddressInfo(peer, 'powValue', data['random'])
+    daemon.decrementThreadCount('announce_node')
+    return retData

onionr/communicatorutils/connectnewpeers.py

@@ -0,0 +1,89 @@
+'''
+    Onionr - Private P2P Communication
+
+    Connect a new peer to our communicator instance. Does so randomly if no peer is specified
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import time, sys, secrets
+import onionrexceptions, logger, onionrpeers
+from utils import networkmerger
+from onionrutils import stringvalidators, epoch
+from communicator import peeraction, bootstrappeers
+
+def connect_new_peer_to_communicator(comm_inst, peer='', useBootstrap=False):
+    config = comm_inst._core.config
+    retData = False
+    tried = comm_inst.offlinePeers
+    if peer != '':
+        if stringvalidators.validate_transport(peer):
+            peerList = [peer]
+        else:
+            raise onionrexceptions.InvalidAddress('Will not attempt connection test to invalid address')
+    else:
+        peerList = comm_inst._core.listAdders()
+
+    mainPeerList = comm_inst._core.listAdders()
+    peerList = onionrpeers.get_score_sorted_peer_list(comm_inst._core)
+
+    # If we don't have enough peers connected or random chance, select new peers to try
+    if len(peerList) < 8 or secrets.randbelow(4) == 3:
+        tryingNew = []
+        for x in comm_inst.newPeers:
+            if x not in peerList:
+                peerList.append(x)
+                tryingNew.append(x)
+        for i in tryingNew:
+            comm_inst.newPeers.remove(i)
+    
+    if len(peerList) == 0 or useBootstrap:
+        # Avoid duplicating bootstrap addresses in peerList
+        bootstrappeers.add_bootstrap_list_to_peer_list(comm_inst, peerList)
+
+    for address in peerList:
+        if not config.get('tor.v3onions') and len(address) == 62:
+            continue
+        # Don't connect to our own address
+        if address == comm_inst._core.hsAddress:
+            continue
+        # Don't connect to invalid address or if its already been tried/connected, or if its cooled down
+        if len(address) == 0 or address in tried or address in comm_inst.onlinePeers or address in comm_inst.cooldownPeer:
+            continue
+        if comm_inst.shutdown:
+            return
+        # Ping a peer,
+        if peeraction.peer_action(comm_inst, address, 'ping') == 'pong!':
+            time.sleep(0.1)
+            if address not in mainPeerList:
+                # Add a peer to our list if it isn't already since it successfully connected
+                networkmerger.mergeAdders(address, comm_inst._core)
+            if address not in comm_inst.onlinePeers:
+                logger.info('Connected to ' + address, terminal=True)
+                comm_inst.onlinePeers.append(address)
+                comm_inst.connectTimes[address] = epoch.get_epoch()
+            retData = address
+
+            # add peer to profile list if they're not in it
+            for profile in comm_inst.peerProfiles:
+                if profile.address == address:
+                    break
+            else:
+                comm_inst.peerProfiles.append(onionrpeers.PeerProfiles(address, comm_inst._core))
+            break
+        else:
+            # Mark a peer as tried if they failed to respond to ping
+            tried.append(address)
+            logger.debug('Failed to connect to ' + address)
+    return retData

onionr/communicatorutils/cooldownpeer.py

@@ -0,0 +1,53 @@
+'''
+    Onionr - Private P2P Communication
+
+    Select a random online peer in a communicator instance and have them "cool down"
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+from onionrutils import epoch
+from communicator import onlinepeers
+def cooldown_peer(comm_inst):
+    '''Randomly add an online peer to cooldown, so we can connect a new one'''
+    onlinePeerAmount = len(comm_inst.onlinePeers)
+    minTime = 300
+    cooldownTime = 600
+    toCool = ''
+    tempConnectTimes = dict(comm_inst.connectTimes)
+
+    # Remove peers from cooldown that have been there long enough
+    tempCooldown = dict(comm_inst.cooldownPeer)
+    for peer in tempCooldown:
+        if (epoch.get_epoch() - tempCooldown[peer]) >= cooldownTime:
+            del comm_inst.cooldownPeer[peer]
+
+    # Cool down a peer, if we have max connections alive for long enough
+    if onlinePeerAmount >= comm_inst._core.config.get('peers.max_connect', 10, save = True):
+        finding = True
+
+        while finding:
+            try:
+                toCool = min(tempConnectTimes, key=tempConnectTimes.get)
+                if (epoch.get_epoch() - tempConnectTimes[toCool]) < minTime:
+                    del tempConnectTimes[toCool]
+                else:
+                    finding = False
+            except ValueError:
+                break
+        else:
+            onlinepeers.remove_online_peer(comm_inst, toCool)
+            comm_inst.cooldownPeer[toCool] = epoch.get_epoch()
+
+    comm_inst.decrementThreadCount('cooldown_peer')

onionr/communicatorutils/daemonqueuehandler.py

@@ -0,0 +1,56 @@
+'''
+    Onionr - P2P Anonymous Storage Network
+
+    Handle daemon queue commands in the communicator
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import logger
+import onionrevents as events
+from onionrutils import localcommand
+def handle_daemon_commands(comm_inst):
+    cmd = comm_inst._core.daemonQueue()
+    response = ''
+    if cmd is not False:
+        events.event('daemon_command', onionr = comm_inst._core.onionrInst, data = {'cmd' : cmd})
+        if cmd[0] == 'shutdown':
+            comm_inst.shutdown = True
+        elif cmd[0] == 'announceNode':
+            if len(comm_inst.onlinePeers) > 0:
+                comm_inst.announce(cmd[1])
+            else:
+                logger.debug("No nodes connected. Will not introduce node.")
+        elif cmd[0] == 'runCheck': # deprecated
+            logger.debug('Status check; looks good.')
+            open(comm_inst._core.dataDir + '.runcheck', 'w+').close()
+        elif cmd[0] == 'connectedPeers':
+            response = '\n'.join(list(comm_inst.onlinePeers)).strip()
+            if response == '':
+                response = 'none'
+        elif cmd[0] == 'localCommand':
+            response = localcommand.local_command(comm_inst._core, cmd[1])
+        elif cmd[0] == 'pex':
+            for i in comm_inst.timers:
+                if i.timerFunction.__name__ == 'lookupAdders':
+                    i.count = (i.frequency - 1)
+        elif cmd[0] == 'uploadBlock':
+            comm_inst.blocksToUpload.append(cmd[1])
+
+        if cmd[0] not in ('', None):
+            if response != '':
+                localcommand.local_command(comm_inst._core, 'queueResponseAdd/' + cmd[4], post=True, postData={'data': response})
+        response = ''
+
+    comm_inst.decrementThreadCount('handle_daemon_commands')

onionr/communicatorutils/deniableinserts.py

@@ -0,0 +1,31 @@
+'''
+    Onionr - Private P2P Communication
+
+    Use the communicator to insert fake mail messages
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import secrets
+from etc import onionrvalues
+def insert_deniable_block(comm_inst):
+    '''Insert a fake block in order to make it more difficult to track real blocks'''
+    fakePeer = ''
+    chance = 10
+    if secrets.randbelow(chance) == (chance - 1):
+        # This assumes on the libsodium primitives to have key-privacy
+        fakePeer = onionrvalues.DENIABLE_PEER_ADDRESS
+        data = secrets.token_hex(secrets.randbelow(1024) + 1)
+        comm_inst._core.insertBlock(data, header='pm', encryptType='asym', asymPeer=fakePeer, disableForward=True, meta={'subject': 'foo'})
+    comm_inst.decrementThreadCount('insert_deniable_block')

onionr/communicatorutils/downloadblocks/__init__.py

@@ -0,0 +1,116 @@
+'''
+    Onionr - Private P2P Communication
+
+    Download blocks using the communicator instance
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import communicator, onionrexceptions
+import logger, onionrpeers
+from onionrutils import blockmetadata, stringvalidators, validatemetadata
+from . import shoulddownload
+from communicator import peeraction, onlinepeers
+
+def download_blocks_from_communicator(comm_inst):
+    assert isinstance(comm_inst, communicator.OnionrCommunicatorDaemon)
+    for blockHash in list(comm_inst.blockQueue):
+        if len(comm_inst.onlinePeers) == 0:
+            break
+        triedQueuePeers = [] # List of peers we've tried for a block
+        try:
+            blockPeers = list(comm_inst.blockQueue[blockHash])
+        except KeyError:
+            blockPeers = []
+        removeFromQueue = True
+
+        if not shoulddownload.should_download(comm_inst, blockHash):
+            continue
+
+        if comm_inst.shutdown or not comm_inst.isOnline or comm_inst._core.storage_counter.isFull():
+            # Exit loop if shutting down or offline, or disk allocation reached
+            break
+        # Do not download blocks being downloaded
+        if blockHash in comm_inst.currentDownloading:
+            #logger.debug('Already downloading block %s...' % blockHash)
+            continue
+
+        comm_inst.currentDownloading.append(blockHash) # So we can avoid concurrent downloading in other threads of same block
+        if len(blockPeers) == 0:
+            peerUsed = onlinepeers.pick_online_peer(comm_inst)
+        else:
+            blockPeers = comm_inst._core._crypto.randomShuffle(blockPeers)
+            peerUsed = blockPeers.pop(0)
+
+        if not comm_inst.shutdown and peerUsed.strip() != '':
+            logger.info("Attempting to download %s from %s..." % (blockHash[:12], peerUsed))
+        content = peeraction.peer_action(comm_inst, peerUsed, 'getdata/' + blockHash, max_resp_size=3000000) # block content from random peer (includes metadata)
+        if content != False and len(content) > 0:
+            try:
+                content = content.encode()
+            except AttributeError:
+                pass
+
+            realHash = comm_inst._core._crypto.sha3Hash(content)
+            try:
+                realHash = realHash.decode() # bytes on some versions for some reason
+            except AttributeError:
+                pass
+            if realHash == blockHash:
+                content = content.decode() # decode here because sha3Hash needs bytes above
+                metas = blockmetadata.get_block_metadata_from_data(content) # returns tuple(metadata, meta), meta is also in metadata
+                metadata = metas[0]
+                if validatemetadata.validate_metadata(comm_inst._core, metadata, metas[2]): # check if metadata is valid, and verify nonce
+                    if comm_inst._core._crypto.verifyPow(content): # check if POW is enough/correct
+                        logger.info('Attempting to save block %s...' % blockHash[:12])
+                        try:
+                            comm_inst._core.setData(content)
+                        except onionrexceptions.DataExists:
+                            logger.warn('Data is already set for %s ' % (blockHash,))
+                        except onionrexceptions.DiskAllocationReached:
+                            logger.error('Reached disk allocation allowance, cannot save block %s.' % (blockHash,))
+                            removeFromQueue = False
+                        else:
+                            comm_inst._core.addToBlockDB(blockHash, dataSaved=True)
+                            blockmetadata.process_block_metadata(comm_inst._core, blockHash) # caches block metadata values to block database
+                    else:
+                        logger.warn('POW failed for block %s.' % (blockHash,))
+                else:
+                    if comm_inst._core._blacklist.inBlacklist(realHash):
+                        logger.warn('Block %s is blacklisted.' % (realHash,))
+                    else:
+                        logger.warn('Metadata for block %s is invalid.' % (blockHash,))
+                        comm_inst._core._blacklist.addToDB(blockHash)
+            else:
+                # if block didn't meet expected hash
+                tempHash = comm_inst._core._crypto.sha3Hash(content) # lazy hack, TODO use var
+                try:
+                    tempHash = tempHash.decode()
+                except AttributeError:
+                    pass
+                # Punish peer for sharing invalid block (not always malicious, but is bad regardless)
+                onionrpeers.PeerProfiles(peerUsed, comm_inst._core).addScore(-50)
+                if tempHash != 'ed55e34cb828232d6c14da0479709bfa10a0923dca2b380496e6b2ed4f7a0253':
+                    # Dumb hack for 404 response from peer. Don't log it if 404 since its likely not malicious or a critical error.
+                    logger.warn('Block hash validation failed for ' + blockHash + ' got ' + tempHash)
+                else:
+                    removeFromQueue = False # Don't remove from queue if 404
+            if removeFromQueue:
+                try:
+                    del comm_inst.blockQueue[blockHash] # remove from block queue both if success or false
+                    logger.info('%s blocks remaining in queue' % [len(comm_inst.blockQueue)], terminal=True)
+                except KeyError:
+                    pass
+        comm_inst.currentDownloading.remove(blockHash)
+    comm_inst.decrementThreadCount('getBlocks')

onionr/communicatorutils/downloadblocks/shoulddownload.py

@@ -0,0 +1,33 @@
+'''
+    Onionr - Private P2P Communication
+
+    Check if a block should be downloaded (if we already have it or its blacklisted or not)
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+def should_download(comm_inst, block_hash):
+    ret_data = True
+    if block_hash in comm_inst._core.getBlockList(): # Dont download block we have
+        ret_data = False
+    else:
+        if comm_inst._core._blacklist.inBlacklist(block_hash): # Dont download blacklisted block
+            ret_data = False
+    if ret_data is False:
+        # Remove block from communicator queue if it shouldnt be downloaded
+        try:
+            del comm_inst.blockQueue[block_hash]
+        except KeyError:
+            pass
+    return ret_data

onionr/communicatorutils/housekeeping.py

@@ -0,0 +1,60 @@
+'''
+    Onionr - Private P2P Communication
+
+    Cleanup old Onionr blocks and forward secrecy keys using the communicator. Ran from a timer usually
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sqlite3
+import logger
+from onionrusers import onionrusers
+from onionrutils import epoch
+def clean_old_blocks(comm_inst):
+    '''Delete old blocks if our disk allocation is full/near full, and also expired blocks'''
+
+    # Delete expired blocks
+    for bHash in comm_inst._core.getExpiredBlocks():
+        comm_inst._core._blacklist.addToDB(bHash)
+        comm_inst._core.removeBlock(bHash)
+        logger.info('Deleted block: %s' % (bHash,))
+
+    while comm_inst._core.storage_counter.isFull():
+        oldest = comm_inst._core.getBlockList()[0]
+        comm_inst._core._blacklist.addToDB(oldest)
+        comm_inst._core.removeBlock(oldest)
+        logger.info('Deleted block: %s' % (oldest,))
+
+    comm_inst.decrementThreadCount('clean_old_blocks')
+
+def clean_keys(comm_inst):
+    '''Delete expired forward secrecy keys'''
+    conn = sqlite3.connect(comm_inst._core.peerDB, timeout=10)
+    c = conn.cursor()
+    time = epoch.get_epoch()
+    deleteKeys = []
+
+    for entry in c.execute("SELECT * FROM forwardKeys WHERE expire <= ?", (time,)):
+        logger.debug('Forward key: %s' % entry[1])
+        deleteKeys.append(entry[1])
+
+    for key in deleteKeys:
+        logger.debug('Deleting forward key %s' % key)
+        c.execute("DELETE from forwardKeys where forwardKey = ?", (key,))
+    conn.commit()
+    conn.close()
+
+    onionrusers.deleteExpiredKeys(comm_inst._core)
+
+    comm_inst.decrementThreadCount('clean_keys')

onionr/communicatorutils/lookupadders.py

@@ -0,0 +1,49 @@
+'''
+    Onionr - Private P2P Communication
+
+    Lookup new peer transport addresses using the communicator
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import logger
+from onionrutils import stringvalidators
+from communicator import peeraction, onlinepeers
+def lookup_new_peer_transports_with_communicator(comm_inst):
+    logger.info('Looking up new addresses...')
+    tryAmount = 1
+    newPeers = []
+    for i in range(tryAmount):
+        # Download new peer address list from random online peers
+        if len(newPeers) > 10000:
+            # Don't get new peers if we have too many queued up
+            break
+        peer = onlinepeers.pick_online_peer(comm_inst)
+        newAdders = peeraction.peer_action(comm_inst, peer, action='pex')
+        try:
+            newPeers = newAdders.split(',')
+        except AttributeError:
+            pass
+    else:
+        # Validate new peers are good format and not already in queue
+        invalid = []
+        for x in newPeers:
+            x = x.strip()
+            if not stringvalidators.validate_transport(x) or x in comm_inst.newPeers or x == comm_inst._core.hsAddress:
+                # avoid adding if its our address
+                invalid.append(x)
+        for x in invalid:
+            newPeers.remove(x)
+        comm_inst.newPeers.extend(newPeers)
+    comm_inst.decrementThreadCount('lookupAdders')

onionr/communicatorutils/lookupblocks.py

@@ -0,0 +1,89 @@
+'''
+    Onionr - Private P2P Communication
+
+    Lookup new blocks with the communicator using a random connected peer
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import logger, onionrproofs
+from onionrutils import stringvalidators, epoch
+from communicator import peeraction, onlinepeers
+
+def lookup_blocks_from_communicator(comm_inst):
+    logger.info('Looking up new blocks...')
+    tryAmount = 2
+    newBlocks = ''
+    existingBlocks = comm_inst._core.getBlockList()
+    triedPeers = [] # list of peers we've tried this time around
+    maxBacklog = 1560 # Max amount of *new* block hashes to have already in queue, to avoid memory exhaustion
+    lastLookupTime = 0 # Last time we looked up a particular peer's list
+    new_block_count = 0
+    for i in range(tryAmount):
+        listLookupCommand = 'getblocklist' # This is defined here to reset it each time
+        if len(comm_inst.blockQueue) >= maxBacklog:
+            break
+        if not comm_inst.isOnline:
+            break
+        # check if disk allocation is used
+        if comm_inst._core.storage_counter.isFull():
+            logger.debug('Not looking up new blocks due to maximum amount of allowed disk space used')
+            break
+        peer = onlinepeers.pick_online_peer(comm_inst) # select random online peer
+        # if we've already tried all the online peers this time around, stop
+        if peer in triedPeers:
+            if len(comm_inst.onlinePeers) == len(triedPeers):
+                break
+            else:
+                continue
+        triedPeers.append(peer)
+
+        # Get the last time we looked up a peer's stamp to only fetch blocks since then.
+        # Saved in memory only for privacy reasons
+        try:
+            lastLookupTime = comm_inst.dbTimestamps[peer]
+        except KeyError:
+            lastLookupTime = 0
+        else:
+            listLookupCommand += '?date=%s' % (lastLookupTime,)
+        try:
+            newBlocks = peeraction.peer_action(comm_inst, peer, listLookupCommand) # get list of new block hashes
+        except Exception as error:
+            logger.warn('Could not get new blocks from %s.' % peer, error = error)
+            newBlocks = False
+        else:
+            comm_inst.dbTimestamps[peer] = epoch.get_rounded_epoch(roundS=60)
+        if newBlocks != False:
+            # if request was a success
+            for i in newBlocks.split('\n'):
+                if stringvalidators.validate_hash(i):
+                    # if newline seperated string is valid hash
+                    if not i in existingBlocks:
+                        # if block does not exist on disk and is not already in block queue
+                        if i not in comm_inst.blockQueue:
+                            if onionrproofs.hashMeetsDifficulty(i) and not comm_inst._core._blacklist.inBlacklist(i):
+                                if len(comm_inst.blockQueue) <= 1000000:
+                                    comm_inst.blockQueue[i] = [peer] # add blocks to download queue
+                                    new_block_count += 1
+                        else:
+                            if peer not in comm_inst.blockQueue[i]:
+                                if len(comm_inst.blockQueue[i]) < 10:
+                                    comm_inst.blockQueue[i].append(peer)
+    if new_block_count > 0:
+        block_string = ""
+        if new_block_count > 1:
+            block_string = "s"
+        logger.info('Discovered %s new block%s' % (new_block_count, block_string), terminal=True)
+    comm_inst.decrementThreadCount('lookupBlocks')
+    return

onionr/communicatorutils/netcheck.py

@@ -0,0 +1,41 @@
+'''
+    Onionr - Private P2P Communication
+
+    Determine if our node is able to use Tor based on the status of a communicator instance
+    and the result of pinging onion http servers
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import logger
+from utils import netutils
+from onionrutils import localcommand, epoch
+def net_check(comm_inst):
+    '''Check if we are connected to the internet or not when we can't connect to any peers'''
+    rec = False # for detecting if we have received incoming connections recently
+    c = comm_inst._core
+    if len(comm_inst.onlinePeers) == 0:
+        try:
+            if (epoch.get_epoch() - int(localcommand.local_command(c, '/lastconnect'))) <= 60:
+                comm_inst.isOnline = True
+                rec = True
+        except ValueError:
+            pass
+        if not rec and not netutils.checkNetwork(c, torPort=comm_inst.proxyPort):
+            if not comm_inst.shutdown:
+                logger.warn('Network check failed, are you connected to the Internet, and is Tor working?')
+            comm_inst.isOnline = False
+        else:
+            comm_inst.isOnline = True
+    comm_inst.decrementThreadCount('net_check')

onionr/communicatorutils/onionrcommunicatortimers.py

@@ -1,6 +1,5 @@
-#!/usr/bin/env python3
 '''
-    Onionr - P2P Anonymous Storage Network
+    Onionr - Private P2P Communication
 
     This file contains timer control for the communicator
 '''
@@ -20,7 +19,7 @@
 '''
 import threading, onionrexceptions, logger
 class OnionrCommunicatorTimers:
-    def __init__(self, daemonInstance, timerFunction, frequency, makeThread=True, threadAmount=1, maxThreads=5, requiresPeer=False):
+    def __init__(self, daemonInstance, timerFunction, frequency, makeThread=True, threadAmount=1, maxThreads=5, requiresPeer=False, myArgs=[]):
         self.timerFunction = timerFunction
         self.frequency = frequency
         self.threadAmount = threadAmount
@@ -29,6 +28,7 @@ class OnionrCommunicatorTimers:
         self.daemonInstance = daemonInstance
         self.maxThreads = maxThreads
         self._core = self.daemonInstance._core
+        self.args = myArgs
 
         self.daemonInstance.timers.append(self)
         self.count = 0
@@ -52,10 +52,10 @@ class OnionrCommunicatorTimers:
                 if self.makeThread:
                     for i in range(self.threadAmount):
                         if self.daemonInstance.threadCounts[self.timerFunction.__name__] >= self.maxThreads:
-                            logger.debug('%s is currently using the maximum number of threads, not starting another.' % self.timerFunction.__name__)
+                            logger.debug('%s is currently using the maximum number of threads, not starting another.' % self.timerFunction.__name__, terminal=True)
                         else:
                             self.daemonInstance.threadCounts[self.timerFunction.__name__] += 1
-                            newThread = threading.Thread(target=self.timerFunction)
+                            newThread = threading.Thread(target=self.timerFunction, args=self.args, daemon=True)
                             newThread.start()
                 else:
                     self.timerFunction()

onionr/communicatorutils/onionrdaemontools.py

@@ -1,214 +0,0 @@
-'''
-    Onionr - P2P Anonymous Storage Network
-
-    Contains the CommunicatorUtils class which contains useful functions for the communicator daemon
-'''
-'''
-    This program is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation, either version 3 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License
-    along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
-
-import onionrexceptions, onionrpeers, onionrproofs, logger
-import base64, sqlite3, os
-from dependencies import secrets
-from utils import netutils
-from onionrusers import onionrusers
-
-class DaemonTools:
-    '''
-        Class intended for use by Onionr Communicator
-    '''
-    def __init__(self, daemon):
-            self.daemon = daemon
-            self.announceProgress = {}
-            self.announceCache = {}
-
-    def announceNode(self):
-        '''Announce our node to our peers'''
-        retData = False
-        announceFail = False
-        
-        # Do not let announceCache get too large
-        if len(self.announceCache) >= 10000:
-            self.announceCache.popitem()
-
-        if self.daemon._core.config.get('general.security_level', 0) == 0:
-            # Announce to random online peers
-            for i in self.daemon.onlinePeers:
-                if not i in self.announceCache and not i in self.announceProgress:
-                    peer = i
-                    break
-            else:
-                peer = self.daemon.pickOnlinePeer()
-
-            for x in range(1):
-                if x == 1 and self.daemon._core.config.get('i2p.host'):
-                    ourID = self.daemon._core.config.get('i2p.own_addr').strip()
-                else:
-                    ourID = self.daemon._core.hsAddress.strip()
-
-                url = 'http://' + peer + '/announce'
-                data = {'node': ourID}
-
-                combinedNodes = ourID + peer
-                if ourID != 1:
-                    #TODO: Extend existingRand for i2p
-                    existingRand = self.daemon._core.getAddressInfo(peer, 'powValue')
-                    if type(existingRand) is type(None):
-                        existingRand = ''
-
-                if peer in self.announceCache:
-                    data['random'] = self.announceCache[peer]
-                elif len(existingRand) > 0:
-                    data['random'] = existingRand
-                else:
-                    self.announceProgress[peer] = True
-                    proof = onionrproofs.DataPOW(combinedNodes, forceDifficulty=4)
-                    del self.announceProgress[peer]
-                    try:
-                        data['random'] = base64.b64encode(proof.waitForResult()[1])
-                    except TypeError:
-                        # Happens when we failed to produce a proof
-                        logger.error("Failed to produce a pow for announcing to " + peer)
-                        announceFail = True
-                    else:
-                        self.announceCache[peer] = data['random']
-                if not announceFail:
-                    logger.info('Announcing node to ' + url)
-                    if self.daemon._core._utils.doPostRequest(url, data) == 'Success':
-                        logger.info('Successfully introduced node to ' + peer)
-                        retData = True
-                        self.daemon._core.setAddressInfo(peer, 'introduced', 1)
-                        self.daemon._core.setAddressInfo(peer, 'powValue', data['random'])
-        self.daemon.decrementThreadCount('announceNode')
-        return retData
-
-    def netCheck(self):
-        '''Check if we are connected to the internet or not when we can't connect to any peers'''
-        if len(self.daemon.onlinePeers) == 0:
-            if not netutils.checkNetwork(self.daemon._core._utils, torPort=self.daemon.proxyPort):
-                if not self.daemon.shutdown:
-                    logger.warn('Network check failed, are you connected to the internet?')
-                self.daemon.isOnline = False
-            else:
-                self.daemon.isOnline = True
-        self.daemon.decrementThreadCount('netCheck')
-
-    def cleanOldBlocks(self):
-        '''Delete old blocks if our disk allocation is full/near full, and also expired blocks'''
-
-        # Delete expired blocks
-        for bHash in self.daemon._core.getExpiredBlocks():
-            self.daemon._core._blacklist.addToDB(bHash)
-            self.daemon._core.removeBlock(bHash)
-            logger.info('Deleted block: %s' % (bHash,))
-
-        while self.daemon._core._utils.storageCounter.isFull():
-            oldest = self.daemon._core.getBlockList()[0]
-            self.daemon._core._blacklist.addToDB(oldest)
-            self.daemon._core.removeBlock(oldest)
-            logger.info('Deleted block: %s' % (oldest,))
-
-        self.daemon.decrementThreadCount('cleanOldBlocks')
-
-    def cleanKeys(self):
-        '''Delete expired forward secrecy keys'''
-        conn = sqlite3.connect(self.daemon._core.peerDB, timeout=10)
-        c = conn.cursor()
-        time = self.daemon._core._utils.getEpoch()
-        deleteKeys = []
-
-        for entry in c.execute("SELECT * FROM forwardKeys WHERE expire <= ?", (time,)):
-            logger.debug('Forward key: %s' % entry[1])
-            deleteKeys.append(entry[1])
-
-        for key in deleteKeys:
-            logger.debug('Deleting forward key %s' % key)
-            c.execute("DELETE from forwardKeys where forwardKey = ?", (key,))
-        conn.commit()
-        conn.close()
-
-        onionrusers.deleteExpiredKeys(self.daemon._core)
-
-        self.daemon.decrementThreadCount('cleanKeys')
-
-    def cooldownPeer(self):
-        '''Randomly add an online peer to cooldown, so we can connect a new one'''
-        onlinePeerAmount = len(self.daemon.onlinePeers)
-        minTime = 300
-        cooldownTime = 600
-        toCool = ''
-        tempConnectTimes = dict(self.daemon.connectTimes)
-
-        # Remove peers from cooldown that have been there long enough
-        tempCooldown = dict(self.daemon.cooldownPeer)
-        for peer in tempCooldown:
-            if (self.daemon._core._utils.getEpoch() - tempCooldown[peer]) >= cooldownTime:
-                del self.daemon.cooldownPeer[peer]
-
-        # Cool down a peer, if we have max connections alive for long enough
-        if onlinePeerAmount >= self.daemon._core.config.get('peers.max_connect', 10, save = True):
-            finding = True
-
-            while finding:
-                try:
-                    toCool = min(tempConnectTimes, key=tempConnectTimes.get)
-                    if (self.daemon._core._utils.getEpoch() - tempConnectTimes[toCool]) < minTime:
-                        del tempConnectTimes[toCool]
-                    else:
-                        finding = False
-                except ValueError:
-                    break
-            else:
-                self.daemon.removeOnlinePeer(toCool)
-                self.daemon.cooldownPeer[toCool] = self.daemon._core._utils.getEpoch()
-
-        self.daemon.decrementThreadCount('cooldownPeer')
-
-    def runCheck(self):
-        if os.path.isfile(self.daemon._core.dataDir + '.runcheck'):
-            os.remove(self.daemon._core.dataDir + '.runcheck')
-            return True
-
-        return False
-
-    def humanReadableTime(self, seconds):
-        build = ''
-
-        units = {
-            'year' : 31557600,
-            'month' : (31557600 / 12),
-            'day' : 86400,
-            'hour' : 3600,
-            'minute' : 60,
-            'second' : 1
-        }
-
-        for unit in units:
-            amnt_unit = int(seconds / units[unit])
-            if amnt_unit >= 1:
-                seconds -= amnt_unit * units[unit]
-                build += '%s %s' % (amnt_unit, unit) + ('s' if amnt_unit != 1 else '') + ' '
-
-        return build.strip()
-
-    def insertDeniableBlock(self):
-        '''Insert a fake block in order to make it more difficult to track real blocks'''
-        fakePeer = ''
-        chance = 10
-        if secrets.randbelow(chance) == (chance - 1):
-            fakePeer = 'OVPCZLOXD6DC5JHX4EQ3PSOGAZ3T24F75HQLIUZSDSMYPEOXCPFA===='
-            data = secrets.token_hex(secrets.randbelow(500) + 1)
-            self.daemon._core.insertBlock(data, header='pm', encryptType='asym', asymPeer=fakePeer, meta={'subject': 'foo'})
-        self.daemon.decrementThreadCount('insertDeniableBlock')
-        return

onionr/communicatorutils/proxypicker.py

@@ -1,7 +1,7 @@
 '''
-    Onionr - P2P Anonymous Storage Network
+    Onionr - Private P2P Communication
 
-    Just picks a proxy
+    Just picks a proxy to use based on a peer's address
 '''
 '''
     This program is free software: you can redistribute it and/or modify
@@ -22,4 +22,5 @@ def pick_proxy(peer_address):
     if peer_address.endswith('.onion'):
         return 'tor'
     elif peer_address.endswith('.i2p'):
-        return 'i2p'
+        return 'i2p'
+    raise ValueError("Peer address was not string ending with acceptable value")

onionr/communicatorutils/servicecreator.py

@@ -0,0 +1,40 @@
+'''
+    Onionr - Private P2P Communication
+
+    Creates an onionr direct connection service by scanning all connection blocks
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import communicator, onionrblockapi
+from onionrutils import stringvalidators, bytesconverter
+
+def service_creator(daemon):
+    assert isinstance(daemon, communicator.OnionrCommunicatorDaemon)
+    core = daemon._core
+    
+    # Find socket connection blocks
+    # TODO cache blocks and only look at recently received ones
+    con_blocks = core.getBlocksByType('con')
+    for b in con_blocks:
+        if not b in daemon.active_services:
+            bl = onionrblockapi.Block(b, core=core, decrypt=True)
+            bs = bytesconverter.bytes_to_str(bl.bcontent) + '.onion'
+            if stringvalidators.validate_pub_key(bl.signer) and stringvalidators.validate_transport(bs):
+                signer = bytesconverter.bytes_to_str(bl.signer)
+                daemon.active_services.append(b)
+                daemon.active_services.append(signer)
+                daemon.services.create_server(signer, bs)
+    
+    daemon.decrementThreadCount('service_creator')

onionr/communicatorutils/uploadblocks.py

@@ -0,0 +1,57 @@
+'''
+    Onionr - Private P2P Communication
+
+    Upload blocks in the upload queue to peers from the communicator
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import logger
+from communicatorutils import proxypicker
+import onionrblockapi as block
+from onionrutils import localcommand, stringvalidators, basicrequests
+from communicator import onlinepeers
+
+def upload_blocks_from_communicator(comm_inst):
+    # when inserting a block, we try to upload it to a few peers to add some deniability
+    TIMER_NAME = "upload_blocks_from_communicator"
+
+    triedPeers = []
+    finishedUploads = []
+    core = comm_inst._core
+    comm_inst.blocksToUpload = core._crypto.randomShuffle(comm_inst.blocksToUpload)
+    if len(comm_inst.blocksToUpload) != 0:
+        for bl in comm_inst.blocksToUpload:
+            if not stringvalidators.validate_hash(bl):
+                logger.warn('Requested to upload invalid block', terminal=True)
+                comm_inst.decrementThreadCount(TIMER_NAME)
+                return
+            for i in range(min(len(comm_inst.onlinePeers), 6)):
+                peer = onlinepeers.pick_online_peer(comm_inst)
+                if peer in triedPeers:
+                    continue
+                triedPeers.append(peer)
+                url = 'http://' + peer + '/upload'
+                data = {'block': block.Block(bl).getRaw()}
+                proxyType = proxypicker.pick_proxy(peer)
+                logger.info("Uploading block to " + peer, terminal=True)
+                if not basicrequests.do_post_request(core, url, data=data, proxyType=proxyType) == False:
+                    localcommand.local_command(core, 'waitforshare/' + bl, post=True)
+                    finishedUploads.append(bl)
+    for x in finishedUploads:
+        try:
+            comm_inst.blocksToUpload.remove(x)
+        except ValueError:
+            pass
+    comm_inst.decrementThreadCount(TIMER_NAME)

onionr/config.py

@@ -1,5 +1,5 @@
 '''
-    Onionr - P2P Microblogging Platform & Social network
+    Onionr - Private P2P Communication
 
     This file deals with configuration management.
 '''
@@ -19,11 +19,9 @@
 '''
 
 import os, json, logger
-
+from utils import identifyhome
 # set data dir
-dataDir = os.environ.get('ONIONR_HOME', os.environ.get('DATA_DIR', 'data/'))
+dataDir = identifyhome.identify_home()
-if not dataDir.endswith('/'):
-    dataDir += '/'
 
 _configfile = os.path.abspath(dataDir + 'config.json')
 _config = {}
@@ -96,15 +94,8 @@ def check():
         Checks if the configuration file exists, creates it if not
     '''
 
-    try:
+    if not os.path.exists(os.path.dirname(get_config_file())):
-        if not os.path.exists(os.path.dirname(get_config_file())):
+        os.makedirs(os.path.dirname(get_config_file()))
-            os.path.mkdirs(os.path.dirname(get_config_file()))
-        if not os.path.isfile(get_config_file()):
-            open(get_config_file(), 'a', encoding="utf8").close()
-            save()
-    except:
-        pass
-        #logger.debug('Failed to check configuration file.')
 
 def save():
     '''
@@ -115,14 +106,13 @@ def save():
     try:
         with open(get_config_file(), 'w', encoding="utf8") as configfile:
             json.dump(get_config(), configfile, indent=2)
-    except:
+    except json.JSONDecodeError:
         logger.warn('Failed to write to configuration file.')
 
 def reload():
     '''
         Reloads the configuration data in memory from the file
     '''
-
     check()
     try:
         with open(get_config_file(), 'r', encoding="utf8") as configfile:

onionr/core.py

@@ -1,5 +1,5 @@
 '''
-    Onionr - P2P Anonymous Storage Network
+    Onionr - Private P2P Communication
 
     Core Onionr library, useful for external programs. Handles peer & data processing
 '''
@@ -17,34 +17,35 @@
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
-import sqlite3, os, sys, time, math, base64, tarfile, nacl, logger, json, netcontroller, math, config, uuid
+import os, sys, json
+import logger, netcontroller, config
 from onionrblockapi import Block
+import coredb
 import deadsimplekv as simplekv
-import onionrutils, onionrcrypto, onionrproofs, onionrevents as events, onionrexceptions
+import onionrcrypto, onionrproofs, onionrevents as events, onionrexceptions
 import onionrblacklist
 from onionrusers import onionrusers
+from onionrstorage import removeblock, setdata
 import dbcreator, onionrstorage, serializeddata, subprocesspow
-from etc import onionrvalues
+from etc import onionrvalues, powchoice
-
+from onionrutils import localcommand, stringvalidators, bytesconverter, epoch
-if sys.version_info < (3, 6):
+from onionrutils import blockmetadata
-    try:
+from utils import identifyhome
-        import sha3
+import storagecounter
-    except ModuleNotFoundError:
-        logger.fatal('On Python 3 versions prior to 3.6.x, you need the sha3 module')
-        sys.exit(1)
 
 class Core:
     def __init__(self, torPort=0):
         '''
             Initialize Core Onionr library
         '''
-
         # set data dir
-        self.dataDir = os.environ.get('ONIONR_HOME', os.environ.get('DATA_DIR', 'data/'))
+        self.dataDir = identifyhome.identify_home()
-        if not self.dataDir.endswith('/'):
-            self.dataDir += '/'
 
         try:
+            self.usageFile = self.dataDir + 'disk-usage.txt'
+            self.config = config
+            self.maxBlockSize = 10000000 # max block size in bytes
+
             self.onionrInst = None
             self.queueDB = self.dataDir + 'queue.db'
             self.peerDB = self.dataDir + 'peers.db'
@@ -64,7 +65,8 @@ class Core:
             self.dbCreate = dbcreator.DBCreator(self)
             self.forwardKeysFile = self.dataDir + 'forward-keys.db'
             self.keyStore = simplekv.DeadSimpleKV(self.dataDir + 'cachedstorage.dat', refresh_seconds=5)
-
+            self.storage_counter = storagecounter.StorageCounter(self)
+            
             # Socket data, defined here because of multithreading constraints with gevent
             self.killSockets = False
             self.startSocket = {}
@@ -72,11 +74,6 @@ class Core:
             self.socketReasons = {}
             self.socketServerResponseData = {}
 
-            self.usageFile = self.dataDir + 'disk-usage.txt'
-            self.config = config
-
-            self.maxBlockSize = 10000000 # max block size in bytes
-
             if not os.path.exists(self.dataDir):
                 os.mkdir(self.dataDir)
             if not os.path.exists(self.dataDir + 'blocks/'):
@@ -103,15 +100,15 @@ class Core:
             else:
                 logger.warn('Warning: address bootstrap file not found ' + self.bootstrapFileLocation)
 
-            self._utils = onionrutils.OnionrUtils(self)
+            self.use_subprocess = powchoice.use_subprocess(self)
             # Initialize the crypto object
             self._crypto = onionrcrypto.OnionrCrypto(self)
             self._blacklist = onionrblacklist.OnionrBlackList(self)
             self.serializer = serializeddata.SerializedData(self)
 
         except Exception as error:
-            logger.error('Failed to initialize core Onionr library.', error=error)
+            logger.error('Failed to initialize core Onionr library.', error=error, terminal=True)
-            logger.fatal('Cannot recover from error.')
+            logger.fatal('Cannot recover from error.', terminal=True)
             sys.exit(1)
         return
 
@@ -127,88 +124,19 @@ class Core:
         '''
             Adds a public key to the key database (misleading function name)
         '''
-        assert peerID not in self.listPeers()
+        return coredb.keydb.addkeys.add_peer(self, peerID, name)
-
-        # This function simply adds a peer to the DB
-        if not self._utils.validatePubKey(peerID):
-            return False
-
-        events.event('pubkey_add', data = {'key': peerID}, onionr = None)
-
-        conn = sqlite3.connect(self.peerDB, timeout=30)
-        hashID = self._crypto.pubKeyHashID(peerID)
-        c = conn.cursor()
-        t = (peerID, name, 'unknown', hashID, 0)
-
-        for i in c.execute("SELECT * FROM peers WHERE id = ?;", (peerID,)):
-            try:
-                if i[0] == peerID:
-                    conn.close()
-                    return False
-            except ValueError:
-                pass
-            except IndexError:
-                pass
-        c.execute('INSERT INTO peers (id, name, dateSeen, hashID, trust) VALUES(?, ?, ?, ?, ?);', t)
-        conn.commit()
-        conn.close()
-
-        return True
 
     def addAddress(self, address):
         '''
             Add an address to the address database (only tor currently)
         '''
-
+        return coredb.keydb.addkeys.add_address(self, address)
-        if address == config.get('i2p.ownAddr', None) or address == self.hsAddress:
-            return False
-        if type(address) is None or len(address) == 0:
-            return False
-        if self._utils.validateID(address):
-            conn = sqlite3.connect(self.addressDB, timeout=30)
-            c = conn.cursor()
-            # check if address is in database
-            # this is safe to do because the address is validated above, but we strip some chars here too just in case
-            address = address.replace('\'', '').replace(';', '').replace('"', '').replace('\\', '')
-            for i in c.execute("SELECT * FROM adders WHERE address = ?;", (address,)):
-                try:
-                    if i[0] == address:
-                        conn.close()
-                        return False
-                except ValueError:
-                    pass
-                except IndexError:
-                    pass
-
-            t = (address, 1)
-            c.execute('INSERT INTO adders (address, type) VALUES(?, ?);', t)
-            conn.commit()
-            conn.close()
-
-            events.event('address_add', data = {'address': address}, onionr = None)
-
-            return True
-        else:
-            #logger.debug('Invalid ID: %s' % address)
-            return False
 
     def removeAddress(self, address):
         '''
             Remove an address from the address database
         '''
-
+        return coredb.keydb.removekeys.remove_address(self, address)
-        if self._utils.validateID(address):
-            conn = sqlite3.connect(self.addressDB, timeout=30)
-            c = conn.cursor()
-            t = (address,)
-            c.execute('Delete from adders where address=?;', t)
-            conn.commit()
-            conn.close()
-
-            events.event('address_remove', data = {'address': address}, onionr = None)
-            return True
-        else:
-            return False
 
     def removeBlock(self, block):
         '''
@@ -216,16 +144,7 @@ class Core:
 
             **You may want blacklist.addToDB(blockHash)
         '''
-
+        removeblock.remove_block(self, block)
-        if self._utils.validateHash(block):
-            conn = sqlite3.connect(self.blockDB, timeout=30)
-            c = conn.cursor()
-            t = (block,)
-            c.execute('Delete from hashes where hash=?;', t)
-            conn.commit()
-            conn.close()
-            dataSize = sys.getsizeof(onionrstorage.getData(self, block))
-            self._utils.storageCounter.removeBytes(dataSize)
 
     def createAddressDB(self):
         '''
@@ -251,67 +170,19 @@ class Core:
 
             Should be in hex format!
         '''
-
+        coredb.blockmetadb.add.add_to_block_DB(self, newHash, selfInsert, dataSaved)
-        if not os.path.exists(self.blockDB):
-            raise Exception('Block db does not exist')
-        if self._utils.hasBlock(newHash):
-            return
-        conn = sqlite3.connect(self.blockDB, timeout=30)
-        c = conn.cursor()
-        currentTime = self._utils.getEpoch() + self._crypto.secrets.randbelow(301)
-        if selfInsert or dataSaved:
-            selfInsert = 1
-        else:
-            selfInsert = 0
-        data = (newHash, currentTime, '', selfInsert)
-        c.execute('INSERT INTO hashes (hash, dateReceived, dataType, dataSaved) VALUES(?, ?, ?, ?);', data)
-        conn.commit()
-        conn.close()
-
-        return
-
-    def getData(self, hash):
-        '''
-            Simply return the data associated to a hash
-        '''
-
-        data = onionrstorage.getData(self, hash)
-
-        return data
 
     def setData(self, data):
         '''
             Set the data assciated with a hash
         '''
+        return onionrstorage.setdata.set_data(self, data)
 
-        data = data
+    def getData(self, hash):
-        dataSize = sys.getsizeof(data)
+        '''
-
+            Simply return the data associated to a hash
-        if not type(data) is bytes:
+        '''
-            data = data.encode()
+        return onionrstorage.getData(self, hash)
-
-        dataHash = self._crypto.sha3Hash(data)
-
-        if type(dataHash) is bytes:
-            dataHash = dataHash.decode()
-        blockFileName = self.blockDataLocation + dataHash + '.dat'
-        if os.path.exists(blockFileName):
-            pass # TODO: properly check if block is already saved elsewhere
-            #raise Exception("Data is already set for " + dataHash)
-        else:
-            if self._utils.storageCounter.addBytes(dataSize) != False:
-                onionrstorage.store(self, data, blockHash=dataHash)
-                conn = sqlite3.connect(self.blockDB, timeout=30)
-                c = conn.cursor()
-                c.execute("UPDATE hashes SET dataSaved=1 WHERE hash = ?;", (dataHash,))
-                conn.commit()
-                conn.close()
-                with open(self.dataNonceFile, 'a') as nonceFile:
-                    nonceFile.write(dataHash + '\n')
-            else:
-                raise onionrexceptions.DiskAllocationReached
-
-        return dataHash
 
     def daemonQueue(self):
         '''
@@ -319,118 +190,31 @@ class Core:
 
             This function intended to be used by the client. Queue to exchange data between "client" and server.
         '''
-
+        return coredb.daemonqueue.daemon_queue(self)
-        retData = False
-        if not os.path.exists(self.queueDB):
-            self.dbCreate.createDaemonDB()
-        else:
-            conn = sqlite3.connect(self.queueDB, timeout=30)
-            c = conn.cursor()
-            try:
-                for row in c.execute('SELECT command, data, date, min(ID), responseID FROM commands group by id'):
-                    retData = row
-                    break
-            except sqlite3.OperationalError:
-                self.dbCreate.createDaemonDB()
-            else:
-                if retData != False:
-                    c.execute('DELETE FROM commands WHERE id=?;', (retData[3],))
-            conn.commit()
-            conn.close()
-
-        events.event('queue_pop', data = {'data': retData}, onionr = None)
-
-        return retData
 
     def daemonQueueAdd(self, command, data='', responseID=''):
         '''
             Add a command to the daemon queue, used by the communication daemon (communicator.py)
         '''
-
+        return coredb.daemonqueue.daemon_queue_add(self, command, data, responseID)
-        retData = True
-
-        date = self._utils.getEpoch()
-        conn = sqlite3.connect(self.queueDB, timeout=30)
-        c = conn.cursor()
-        t = (command, data, date, responseID)
-
-        try:
-            c.execute('INSERT INTO commands (command, data, date, responseID) VALUES(?, ?, ?, ?)', t)
-            conn.commit()
-        except sqlite3.OperationalError:
-            retData = False
-            self.daemonQueue()
-        events.event('queue_push', data = {'command': command, 'data': data}, onionr = None)
-        conn.close()
-        return retData
 
     def daemonQueueGetResponse(self, responseID=''):
         '''
             Get a response sent by communicator to the API, by requesting to the API
         '''
-        assert len(responseID) > 0
+        return coredb.daemonqueue.daemon_queue_get_response(self, responseID)
-        resp = self._utils.localCommand('queueResponse/' + responseID)
-        return resp
-
-    def daemonQueueWaitForResponse(self, responseID='', checkFreqSecs=1):
-        resp = 'failure'
-        while resp == 'failure':
-            resp = self.daemonQueueGetResponse(responseID)
-            time.sleep(1)
-        return resp
-
-    def daemonQueueSimple(self, command, data='', checkFreqSecs=1):
-        '''
-        A simplified way to use the daemon queue. Will register a command (with optional data) and wait, return the data
-        Not always useful, but saves time + LOC in some cases.
-        This is a blocking function, so be careful.
-        '''
-        responseID = str(uuid.uuid4()) # generate unique response ID
-        self.daemonQueueAdd(command, data=data, responseID=responseID)
-        return self.daemonQueueWaitForResponse(responseID, checkFreqSecs)
 
     def clearDaemonQueue(self):
         '''
             Clear the daemon queue (somewhat dangerous)
         '''
-        conn = sqlite3.connect(self.queueDB, timeout=30)
+        return coredb.daemonqueue.clear_daemon_queue(self)
-        c = conn.cursor()
-
-        try:
-            c.execute('DELETE FROM commands;')
-            conn.commit()
-        except:
-            pass
-
-        conn.close()
-        events.event('queue_clear', onionr = None)
-
-        return
 
     def listAdders(self, randomOrder=True, i2p=True, recent=0):
         '''
             Return a list of addresses
         '''
-        conn = sqlite3.connect(self.addressDB, timeout=30)
+        return coredb.keydb.listkeys.list_adders(self, randomOrder, i2p, recent)
-        c = conn.cursor()
-        if randomOrder:
-            addresses = c.execute('SELECT * FROM adders ORDER BY RANDOM();')
-        else:
-            addresses = c.execute('SELECT * FROM adders;')
-        addressList = []
-        for i in addresses:
-            if len(i[0].strip()) == 0:
-                continue
-            addressList.append(i[0])
-        conn.close()
-        testList = list(addressList) # create new list to iterate
-        for address in testList:
-            try:
-                if recent > 0 and (self._utils.getEpoch() - self.getAddressInfo(address, 'lastConnect')) > recent:
-                    raise TypeError # If there is no last-connected date or it was too long ago, don't add peer to list if recent is not 0
-            except TypeError:
-                addressList.remove(address)
-        return addressList
 
     def listPeers(self, randomOrder=True, getPow=False, trust=0):
         '''
@@ -439,35 +223,7 @@ class Core:
             randomOrder determines if the list should be in a random order
             trust sets the minimum trust to list
         '''
-        conn = sqlite3.connect(self.peerDB, timeout=30)
+        return coredb.keydb.listkeys.list_peers(self, randomOrder, getPow, trust)
-        c = conn.cursor()
-
-        payload = ''
-
-        if trust not in (0, 1, 2):
-            logger.error('Tried to select invalid trust.')
-            return
-
-        if randomOrder:
-            payload = 'SELECT * FROM peers WHERE trust >= ? ORDER BY RANDOM();'
-        else:
-            payload = 'SELECT * FROM peers WHERE trust >= ?;'
-
-        peerList = []
-
-        for i in c.execute(payload, (trust,)):
-            try:
-                if len(i[0]) != 0:
-                    if getPow:
-                        peerList.append(i[0] + '-' + i[1])
-                    else:
-                        peerList.append(i[0])
-            except TypeError:
-                pass
-
-        conn.close()
-
-        return peerList
 
     def getPeerInfo(self, peer, info):
         '''
@@ -480,46 +236,13 @@ class Core:
             trust int           4
             hashID text         5
         '''
-        conn = sqlite3.connect(self.peerDB, timeout=30)
+        return coredb.keydb.userinfo.get_user_info(self, peer, info)
-        c = conn.cursor()
-
-        command = (peer,)
-        infoNumbers = {'id': 0, 'name': 1, 'adders': 2, 'dateSeen': 3, 'trust': 4, 'hashID': 5}
-        info = infoNumbers[info]
-        iterCount = 0
-        retVal = ''
-
-        for row in c.execute('SELECT * FROM peers WHERE id=?;', command):
-            for i in row:
-                if iterCount == info:
-                    retVal = i
-                    break
-                else:
-                    iterCount += 1
-
-        conn.close()
-
-        return retVal
 
     def setPeerInfo(self, peer, key, data):
         '''
             Update a peer for a key
         '''
-
+        return coredb.keydb.userinfo.set_peer_info(self, peer, key, data)
-        conn = sqlite3.connect(self.peerDB, timeout=30)
-        c = conn.cursor()
-
-        command = (data, peer)
-
-        # TODO: validate key on whitelist
-        if key not in ('id', 'name', 'pubkey', 'forwardKey', 'dateSeen', 'trust'):
-            raise Exception("Got invalid database key when setting peer info")
-
-        c.execute('UPDATE peers SET ' + key + ' = ? WHERE id=?', command)
-        conn.commit()
-        conn.close()
-
-        return
 
     def getAddressInfo(self, address, info):
         '''
@@ -536,117 +259,35 @@ class Core:
             trust       8
             introduced  9
         '''
-
+        return coredb.keydb.transportinfo.get_address_info(self, address, info)
-        conn = sqlite3.connect(self.addressDB, timeout=30)
-        c = conn.cursor()
-
-        command = (address,)
-        infoNumbers = {'address': 0, 'type': 1, 'knownPeer': 2, 'speed': 3, 'success': 4, 'powValue': 5, 'failure': 6, 'lastConnect': 7, 'trust': 8, 'introduced': 9}
-        info = infoNumbers[info]
-        iterCount = 0
-        retVal = ''
-
-        for row in c.execute('SELECT * FROM adders WHERE address=?;', command):
-            for i in row:
-                if iterCount == info:
-                    retVal = i
-                    break
-                else:
-                    iterCount += 1
-        conn.close()
-
-        return retVal
 
     def setAddressInfo(self, address, key, data):
         '''
             Update an address for a key
         '''
-
+        return coredb.keydb.transportinfo.set_address_info(self, address, key, data)
-        conn = sqlite3.connect(self.addressDB, timeout=30)
-        c = conn.cursor()
-
-        command = (data, address)
-
-        if key not in ('address', 'type', 'knownPeer', 'speed', 'success', 'failure', 'powValue', 'lastConnect', 'lastConnectAttempt', 'trust', 'introduced'):
-            raise Exception("Got invalid database key when setting address info")
-        else:
-            c.execute('UPDATE adders SET ' + key + ' = ? WHERE address=?', command)
-            conn.commit()
-        conn.close()
-
-        return
 
     def getBlockList(self, dateRec = None, unsaved = False):
         '''
             Get list of our blocks
         '''
-        if dateRec == None:
+        return coredb.blockmetadb.get_block_list(self, dateRec, unsaved)
-            dateRec = 0
-
-        conn = sqlite3.connect(self.blockDB, timeout=30)
-        c = conn.cursor()
-
-        execute = 'SELECT hash FROM hashes WHERE dateReceived >= ? ORDER BY dateReceived ASC;'
-        args = (dateRec,)
-        rows = list()
-        for row in c.execute(execute, args):
-            for i in row:
-                rows.append(i)
-        conn.close()
-        return rows
 
     def getBlockDate(self, blockHash):
         '''
             Returns the date a block was received
         '''
-
+        return coredb.blockmetadb.get_block_date(self, blockHash)
-        conn = sqlite3.connect(self.blockDB, timeout=30)
-        c = conn.cursor()
-
-        execute = 'SELECT dateReceived FROM hashes WHERE hash=?;'
-        args = (blockHash,)
-        for row in c.execute(execute, args):
-            for i in row:
-                return int(i)
-        conn.close()
-        return None
 
     def getBlocksByType(self, blockType, orderDate=True):
         '''
             Returns a list of blocks by the type
         '''
-
+        return coredb.blockmetadb.get_blocks_by_type(self, blockType, orderDate)
-        conn = sqlite3.connect(self.blockDB, timeout=30)
-        c = conn.cursor()
-
-        if orderDate:
-            execute = 'SELECT hash FROM hashes WHERE dataType=? ORDER BY dateReceived;'
-        else:
-            execute = 'SELECT hash FROM hashes WHERE dataType=?;'
-
-        args = (blockType,)
-        rows = list()
-
-        for row in c.execute(execute, args):
-            for i in row:
-                rows.append(i)
-        conn.close()
-        return rows
 
     def getExpiredBlocks(self):
         '''Returns a list of expired blocks'''
-        conn = sqlite3.connect(self.blockDB, timeout=30)
+        return coredb.blockmetadb.expiredblocks.get_expired_blocks(self)
-        c = conn.cursor()
-        date = int(self._utils.getEpoch())
-
-        execute = 'SELECT hash FROM hashes WHERE expire <= %s ORDER BY dateReceived;' % (date,)
-
-        rows = list()
-        for row in c.execute(execute):
-            for i in row:
-                rows.append(i)
-        conn.close()
-        return rows
 
     def updateBlockInfo(self, hash, key, data):
         '''
@@ -663,18 +304,7 @@ class Core:
             dateClaimed  - timestamp claimed inside the block, only as trustworthy as the block author is
             expire       - expire date for a block
         '''
-
+        return coredb.blockmetadb.updateblockinfo.update_block_info(self, hash, key, data)
-        if key not in ('dateReceived', 'decrypted', 'dataType', 'dataFound', 'dataSaved', 'sig', 'author', 'dateClaimed', 'expire'):
-            return False
-
-        conn = sqlite3.connect(self.blockDB, timeout=30)
-        c = conn.cursor()
-        args = (data, hash)
-        c.execute("UPDATE hashes SET " + key + " = ? where hash = ?;", args)
-        conn.commit()
-        conn.close()
-
-        return True
 
     def insertBlock(self, data, header='txt', sign=False, encryptType='', symKey='', asymPeer='', meta = {}, expire=None, disableForward=False):
         '''
@@ -682,15 +312,17 @@ class Core:
             encryptType must be specified to encrypt a block
         '''
         allocationReachedMessage = 'Cannot insert block, disk allocation reached.'
-        if self._utils.storageCounter.isFull():
+        if self.storage_counter.isFull():
             logger.error(allocationReachedMessage)
             return False
         retData = False
 
-        createTime = self._utils.getRoundedEpoch()
+        if type(data) is None:
+            raise ValueError('Data cannot be none')
 
-        # check nonce
+        createTime = epoch.get_epoch()
-        dataNonce = self._utils.bytesToStr(self._crypto.sha3Hash(data))
+
+        dataNonce = bytesconverter.bytes_to_str(self._crypto.sha3Hash(data))
         try:
             with open(self.dataNonceFile, 'r') as nonces:
                 if dataNonce in nonces:
@@ -763,14 +395,18 @@ class Core:
             signature = self._crypto.symmetricEncrypt(signature, key=symKey, returnEncoded=True).decode()
             signer = self._crypto.symmetricEncrypt(signer, key=symKey, returnEncoded=True).decode()
         elif encryptType == 'asym':
-            if self._utils.validatePubKey(asymPeer):
+            if stringvalidators.validate_pub_key(asymPeer):
                 # Encrypt block data with forward secrecy key first, but not meta
                 jsonMeta = json.dumps(meta)
                 jsonMeta = self._crypto.pubKeyEncrypt(jsonMeta, asymPeer, encodedData=True).decode()
                 data = self._crypto.pubKeyEncrypt(data, asymPeer, encodedData=True).decode()
                 signature = self._crypto.pubKeyEncrypt(signature, asymPeer, encodedData=True).decode()
                 signer = self._crypto.pubKeyEncrypt(signer, asymPeer, encodedData=True).decode()
-                onionrusers.OnionrUser(self, asymPeer, saveUser=True)
+                try:
+                    onionrusers.OnionrUser(self, asymPeer, saveUser=True)
+                except ValueError:
+                    # if peer is already known
+                    pass
             else:
                 raise onionrexceptions.InvalidPubkey(asymPeer + ' is not a valid base32 encoded ed25519 key')
 
@@ -786,7 +422,10 @@ class Core:
             metadata['expire'] = expire
 
         # send block data (and metadata) to POW module to get tokenized block data
-        payload = subprocesspow.SubprocessPOW(data, metadata, self).start()
+        if self.use_subprocess:
+            payload = subprocesspow.SubprocessPOW(data, metadata, self).start()
+        else:
+            payload = onionrproofs.POW(metadata, data).waitForResult()
         if payload != False:
             try:
                 retData = self.setData(payload)
@@ -794,45 +433,29 @@ class Core:
                 logger.error(allocationReachedMessage)
                 retData = False
             else:
-                # Tell the api server through localCommand to wait for the daemon to upload this block to make stastical analysis more difficult
+                # Tell the api server through localCommand to wait for the daemon to upload this block to make statistical analysis more difficult
-                self._utils.localCommand('/waitforshare/' + retData, post=True)
+                if localcommand.local_command(self, '/ping', maxWait=10) == 'pong!':
+                    if self.config.get('general.security_level', 1) == 0:
+                        localcommand.local_command(self, '/waitforshare/' + retData, post=True, maxWait=5)
+                    self.daemonQueueAdd('uploadBlock', retData)
+                else:
+                    pass
                 self.addToBlockDB(retData, selfInsert=True, dataSaved=True)
-                #self.setBlockType(retData, meta['type'])
+                blockmetadata.process_block_metadata(self, retData)
-                self._utils.processBlockMetadata(retData)
-                self.daemonQueueAdd('uploadBlock', retData)
 
         if retData != False:
-            if plaintextPeer == 'OVPCZLOXD6DC5JHX4EQ3PSOGAZ3T24F75HQLIUZSDSMYPEOXCPFA====':
+            if plaintextPeer == onionrvalues.DENIABLE_PEER_ADDRESS:
-                events.event('insertdeniable', {'content': plaintext, 'meta': plaintextMeta, 'hash': retData, 'peer': self._utils.bytesToStr(asymPeer)}, onionr = self.onionrInst, threaded = True)
+                events.event('insertdeniable', {'content': plaintext, 'meta': plaintextMeta, 'hash': retData, 'peer': bytesconverter.bytes_to_str(asymPeer)}, onionr = self.onionrInst, threaded = True)
             else:
-                events.event('insertblock', {'content': plaintext, 'meta': plaintextMeta, 'hash': retData, 'peer': self._utils.bytesToStr(asymPeer)}, onionr = self.onionrInst, threaded = True)
+                events.event('insertblock', {'content': plaintext, 'meta': plaintextMeta, 'hash': retData, 'peer': bytesconverter.bytes_to_str(asymPeer)}, onionr = self.onionrInst, threaded = True)
         return retData
 
     def introduceNode(self):
         '''
             Introduces our node into the network by telling X many nodes our HS address
         '''
-
+        if localcommand.local_command(self, '/ping', maxWait=10) == 'pong!':
-        if(self._utils.isCommunicatorRunning(timeout=30)):
+            self.daemonQueueAdd('announceNode')
-            announceAmount = 2
+            logger.info('Introduction command will be processed.', terminal=True)
-            nodeList = self.listAdders()
-
-            if len(nodeList) == 0:
-                for i in self.bootstrapList:
-                    if self._utils.validateID(i):
-                        self.addAddress(i)
-                        nodeList.append(i)
-
-            if announceAmount > len(nodeList):
-                announceAmount = len(nodeList)
-
-            for i in range(announceAmount):
-                self.daemonQueueAdd('announceNode', nodeList[i])
-
-            events.event('introduction', onionr = None)
-
-            return True
         else:
-            logger.error('Onionr daemon is not running.')
+            logger.warn('No running node detected. Cannot introduce.', terminal=True)
-            return False
-        return

onionr/coredb/__init__.py

@@ -0,0 +1 @@
+from . import keydb, blockmetadb, daemonqueue

onionr/coredb/blockmetadb/__init__.py

@@ -0,0 +1,77 @@
+'''
+    Onionr - Private P2P Communication
+
+    This module works with information relating to blocks stored on the node
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sqlite3
+from . import expiredblocks, updateblockinfo, add
+def get_block_list(core_inst, dateRec = None, unsaved = False):
+    '''
+        Get list of our blocks
+    '''
+    if dateRec == None:
+        dateRec = 0
+
+    conn = sqlite3.connect(core_inst.blockDB, timeout=30)
+    c = conn.cursor()
+
+    execute = 'SELECT hash FROM hashes WHERE dateReceived >= ? ORDER BY dateReceived ASC;'
+    args = (dateRec,)
+    rows = list()
+    for row in c.execute(execute, args):
+        for i in row:
+            rows.append(i)
+    conn.close()
+    return rows
+
+def get_block_date(core_inst, blockHash):
+    '''
+        Returns the date a block was received
+    '''
+
+    conn = sqlite3.connect(core_inst.blockDB, timeout=30)
+    c = conn.cursor()
+
+    execute = 'SELECT dateReceived FROM hashes WHERE hash=?;'
+    args = (blockHash,)
+    for row in c.execute(execute, args):
+        for i in row:
+            return int(i)
+    conn.close()
+    return None
+
+def get_blocks_by_type(core_inst, blockType, orderDate=True):
+    '''
+        Returns a list of blocks by the type
+    '''
+
+    conn = sqlite3.connect(core_inst.blockDB, timeout=30)
+    c = conn.cursor()
+
+    if orderDate:
+        execute = 'SELECT hash FROM hashes WHERE dataType=? ORDER BY dateReceived;'
+    else:
+        execute = 'SELECT hash FROM hashes WHERE dataType=?;'
+
+    args = (blockType,)
+    rows = list()
+
+    for row in c.execute(execute, args):
+        for i in row:
+            rows.append(i)
+    conn.close()
+    return rows

onionr/coredb/blockmetadb/add.py

@@ -0,0 +1,43 @@
+'''
+    Onionr - Private P2P Communication
+
+    Add an entry to the block metadata database
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import os, sqlite3
+from onionrutils import epoch, blockmetadata
+def add_to_block_DB(core_inst, newHash, selfInsert=False, dataSaved=False):
+    '''
+        Add a hash value to the block db
+
+        Should be in hex format!
+    '''
+
+    if not os.path.exists(core_inst.blockDB):
+        raise Exception('Block db does not exist')
+    if blockmetadata.has_block(core_inst, newHash):
+        return
+    conn = sqlite3.connect(core_inst.blockDB, timeout=30)
+    c = conn.cursor()
+    currentTime = epoch.get_epoch() + core_inst._crypto.secrets.randbelow(301)
+    if selfInsert or dataSaved:
+        selfInsert = 1
+    else:
+        selfInsert = 0
+    data = (newHash, currentTime, '', selfInsert)
+    c.execute('INSERT INTO hashes (hash, dateReceived, dataType, dataSaved) VALUES(?, ?, ?, ?);', data)
+    conn.commit()
+    conn.close()

onionr/coredb/blockmetadb/expiredblocks.py

@@ -0,0 +1,35 @@
+'''
+    Onionr - Private P2P Communication
+
+    Get a list of expired blocks still stored
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sqlite3
+from onionrutils import epoch
+def get_expired_blocks(core_inst):
+    '''Returns a list of expired blocks'''
+    conn = sqlite3.connect(core_inst.blockDB, timeout=30)
+    c = conn.cursor()
+    date = int(epoch.get_epoch())
+
+    execute = 'SELECT hash FROM hashes WHERE expire <= %s ORDER BY dateReceived;' % (date,)
+
+    rows = list()
+    for row in c.execute(execute):
+        for i in row:
+            rows.append(i)
+    conn.close()
+    return rows

onionr/coredb/blockmetadb/updateblockinfo.py

@@ -0,0 +1,33 @@
+'''
+    Onionr - Private P2P Communication
+
+    Update block information in the metadata database by a field name
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+
+import sqlite3
+def update_block_info(core_inst, hash, key, data):
+    if key not in ('dateReceived', 'decrypted', 'dataType', 'dataFound', 'dataSaved', 'sig', 'author', 'dateClaimed', 'expire'):
+        return False
+
+    conn = sqlite3.connect(core_inst.blockDB, timeout=30)
+    c = conn.cursor()
+    args = (data, hash)
+    c.execute("UPDATE hashes SET " + key + " = ? where hash = ?;", args)
+    conn.commit()
+    conn.close()
+
+    return True

onionr/coredb/daemonqueue/__init__.py

@@ -0,0 +1,97 @@
+'''
+    Onionr - Private P2P Communication
+
+    Write and read the daemon queue, which is how messages are passed into the onionr daemon in a more
+    direct way than the http api
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sqlite3, os
+import onionrevents as events
+from onionrutils import localcommand, epoch
+
+def daemon_queue(core_inst):
+    '''
+        Gives commands to the communication proccess/daemon by reading an sqlite3 database
+
+        This function intended to be used by the client. Queue to exchange data between "client" and server.
+    '''
+
+    retData = False
+    if not os.path.exists(core_inst.queueDB):
+        core_inst.dbCreate.createDaemonDB()
+    else:
+        conn = sqlite3.connect(core_inst.queueDB, timeout=30)
+        c = conn.cursor()
+        try:
+            for row in c.execute('SELECT command, data, date, min(ID), responseID FROM commands group by id'):
+                retData = row
+                break
+        except sqlite3.OperationalError:
+            core_inst.dbCreate.createDaemonDB()
+        else:
+            if retData != False:
+                c.execute('DELETE FROM commands WHERE id=?;', (retData[3],))
+        conn.commit()
+        conn.close()
+
+    events.event('queue_pop', data = {'data': retData}, onionr = core_inst.onionrInst)
+
+    return retData
+
+def daemon_queue_add(core_inst, command, data='', responseID=''):
+    '''
+        Add a command to the daemon queue, used by the communication daemon (communicator.py)
+    '''
+
+    retData = True
+
+    date = epoch.get_epoch()
+    conn = sqlite3.connect(core_inst.queueDB, timeout=30)
+    c = conn.cursor()
+    t = (command, data, date, responseID)
+    try:
+        c.execute('INSERT INTO commands (command, data, date, responseID) VALUES(?, ?, ?, ?)', t)
+        conn.commit()
+    except sqlite3.OperationalError:
+        retData = False
+        core_inst.daemonQueue()
+    events.event('queue_push', data = {'command': command, 'data': data}, onionr = core_inst.onionrInst)
+    conn.close()
+    return retData
+
+def daemon_queue_get_response(core_inst, responseID=''):
+    '''
+        Get a response sent by communicator to the API, by requesting to the API
+    '''
+    assert len(responseID) > 0
+    resp = localcommand.local_command(core_inst, 'queueResponse/' + responseID)
+    return resp
+
+def clear_daemon_queue(core_inst):
+    '''
+        Clear the daemon queue (somewhat dangerous)
+    '''
+    conn = sqlite3.connect(core_inst.queueDB, timeout=30)
+    c = conn.cursor()
+
+    try:
+        c.execute('DELETE FROM commands;')
+        conn.commit()
+    except:
+        pass
+
+    conn.close()
+    events.event('queue_clear', onionr = core_inst.onionrInst)

onionr/coredb/keydb/__init__.py

@@ -0,0 +1 @@
+from . import addkeys, listkeys, removekeys, userinfo, transportinfo

onionr/coredb/keydb/addkeys.py

@@ -0,0 +1,91 @@
+'''
+    Onionr - Private P2P Communication
+
+    add user keys or transport addresses
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sqlite3
+import onionrevents as events
+from onionrutils import stringvalidators
+
+def add_peer(core_inst, peerID, name=''):
+    '''
+        Adds a public key to the key database (misleading function name)
+    '''
+    if peerID in core_inst.listPeers() or peerID == core_inst._crypto.pubKey:
+        raise ValueError("specified id is already known")
+
+    # This function simply adds a peer to the DB
+    if not stringvalidators.validate_pub_key(peerID):
+        return False
+
+    events.event('pubkey_add', data = {'key': peerID}, onionr = core_inst.onionrInst)
+
+    conn = sqlite3.connect(core_inst.peerDB, timeout=30)
+    hashID = core_inst._crypto.pubKeyHashID(peerID)
+    c = conn.cursor()
+    t = (peerID, name, 'unknown', hashID, 0)
+
+    for i in c.execute("SELECT * FROM peers WHERE id = ?;", (peerID,)):
+        try:
+            if i[0] == peerID:
+                conn.close()
+                return False
+        except ValueError:
+            pass
+        except IndexError:
+            pass
+    c.execute('INSERT INTO peers (id, name, dateSeen, hashID, trust) VALUES(?, ?, ?, ?, ?);', t)
+    conn.commit()
+    conn.close()
+
+    return True
+
+def add_address(core_inst, address):
+    '''
+        Add an address to the address database (only tor currently)
+    '''
+
+    if type(address) is None or len(address) == 0:
+        return False
+    if stringvalidators.validate_transport(address):
+        if address == core_inst.config.get('i2p.ownAddr', None) or address == core_inst.hsAddress:
+            return False
+        conn = sqlite3.connect(core_inst.addressDB, timeout=30)
+        c = conn.cursor()
+        # check if address is in database
+        # this is safe to do because the address is validated above, but we strip some chars here too just in case
+        address = address.replace('\'', '').replace(';', '').replace('"', '').replace('\\', '')
+        for i in c.execute("SELECT * FROM adders WHERE address = ?;", (address,)):
+            try:
+                if i[0] == address:
+                    conn.close()
+                    return False
+            except ValueError:
+                pass
+            except IndexError:
+                pass
+
+        t = (address, 1)
+        c.execute('INSERT INTO adders (address, type) VALUES(?, ?);', t)
+        conn.commit()
+        conn.close()
+
+        events.event('address_add', data = {'address': address}, onionr = core_inst.onionrInst)
+
+        return True
+    else:
+        return False

onionr/coredb/keydb/listkeys.py

@@ -0,0 +1,83 @@
+'''
+    Onionr - Private P2P Communication
+
+    get lists for user keys or transport addresses
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sqlite3
+import logger
+from onionrutils import epoch
+def list_peers(core_inst, randomOrder=True, getPow=False, trust=0):
+    '''
+        Return a list of public keys (misleading function name)
+
+        randomOrder determines if the list should be in a random order
+        trust sets the minimum trust to list
+    '''
+    conn = sqlite3.connect(core_inst.peerDB, timeout=30)
+    c = conn.cursor()
+
+    payload = ''
+
+    if trust not in (0, 1, 2):
+        logger.error('Tried to select invalid trust.')
+        return
+
+    if randomOrder:
+        payload = 'SELECT * FROM peers WHERE trust >= ? ORDER BY RANDOM();'
+    else:
+        payload = 'SELECT * FROM peers WHERE trust >= ?;'
+
+    peerList = []
+
+    for i in c.execute(payload, (trust,)):
+        try:
+            if len(i[0]) != 0:
+                if getPow:
+                    peerList.append(i[0] + '-' + i[1])
+                else:
+                    peerList.append(i[0])
+        except TypeError:
+            pass
+
+    conn.close()
+
+    return peerList
+
+def list_adders(core_inst, randomOrder=True, i2p=True, recent=0):
+    '''
+        Return a list of transport addresses
+    '''
+    conn = sqlite3.connect(core_inst.addressDB, timeout=30)
+    c = conn.cursor()
+    if randomOrder:
+        addresses = c.execute('SELECT * FROM adders ORDER BY RANDOM();')
+    else:
+        addresses = c.execute('SELECT * FROM adders;')
+    addressList = []
+    for i in addresses:
+        if len(i[0].strip()) == 0:
+            continue
+        addressList.append(i[0])
+    conn.close()
+    testList = list(addressList) # create new list to iterate
+    for address in testList:
+        try:
+            if recent > 0 and (epoch.get_epoch() - core_inst.getAddressInfo(address, 'lastConnect')) > recent:
+                raise TypeError # If there is no last-connected date or it was too long ago, don't add peer to list if recent is not 0
+        except TypeError:
+            addressList.remove(address)
+    return addressList

onionr/coredb/keydb/removekeys.py

@@ -0,0 +1,40 @@
+'''
+    Onionr - Private P2P Communication
+
+    Remove a transport address but don't ban them
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sqlite3
+import onionrevents as events
+from onionrutils import stringvalidators
+
+def remove_address(core_inst, address):
+    '''
+        Remove an address from the address database
+    '''
+
+    if stringvalidators.validate_transport(address):
+        conn = sqlite3.connect(core_inst.addressDB, timeout=30)
+        c = conn.cursor()
+        t = (address,)
+        c.execute('Delete from adders where address=?;', t)
+        conn.commit()
+        conn.close()
+
+        events.event('address_remove', data = {'address': address}, onionr = core_inst.onionrInst)
+        return True
+    else:
+        return False

onionr/coredb/keydb/transportinfo.py

@@ -0,0 +1,72 @@
+'''
+    Onionr - Private P2P Communication
+
+    get or set transport address meta information
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sqlite3
+def get_address_info(core_inst, address, info):
+    '''
+        Get info about an address from its database entry
+
+        address text, 0
+        type int, 1
+        knownPeer text, 2
+        speed int, 3
+        success int, 4
+        powValue    5
+        failure int 6
+        lastConnect 7
+        trust       8
+        introduced  9
+    '''
+
+    conn = sqlite3.connect(core_inst.addressDB, timeout=30)
+    c = conn.cursor()
+
+    command = (address,)
+    infoNumbers = {'address': 0, 'type': 1, 'knownPeer': 2, 'speed': 3, 'success': 4, 'powValue': 5, 'failure': 6, 'lastConnect': 7, 'trust': 8, 'introduced': 9}
+    info = infoNumbers[info]
+    iterCount = 0
+    retVal = ''
+
+    for row in c.execute('SELECT * FROM adders WHERE address=?;', command):
+        for i in row:
+            if iterCount == info:
+                retVal = i
+                break
+            else:
+                iterCount += 1
+    conn.close()
+
+    return retVal
+
+def set_address_info(core_inst, address, key, data):
+    '''
+        Update an address for a key
+    '''
+
+    conn = sqlite3.connect(core_inst.addressDB, timeout=30)
+    c = conn.cursor()
+
+    command = (data, address)
+
+    if key not in ('address', 'type', 'knownPeer', 'speed', 'success', 'failure', 'powValue', 'lastConnect', 'lastConnectAttempt', 'trust', 'introduced'):
+        raise Exception("Got invalid database key when setting address info")
+    else:
+        c.execute('UPDATE adders SET ' + key + ' = ? WHERE address=?', command)
+        conn.commit()
+    conn.close()

onionr/coredb/keydb/userinfo.py

@@ -0,0 +1,69 @@
+'''
+    Onionr - Private P2P Communication
+
+    get or set information about a user id
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sqlite3
+def get_user_info(core_inst, peer, info):
+    '''
+        Get info about a peer from their database entry
+
+        id text             0
+        name text,          1
+        adders text,        2
+        dateSeen not null,  3
+        trust int           4
+        hashID text         5
+    '''
+    conn = sqlite3.connect(core_inst.peerDB, timeout=30)
+    c = conn.cursor()
+
+    command = (peer,)
+    infoNumbers = {'id': 0, 'name': 1, 'adders': 2, 'dateSeen': 3, 'trust': 4, 'hashID': 5}
+    info = infoNumbers[info]
+    iterCount = 0
+    retVal = ''
+
+    for row in c.execute('SELECT * FROM peers WHERE id=?;', command):
+        for i in row:
+            if iterCount == info:
+                retVal = i
+                break
+            else:
+                iterCount += 1
+
+    conn.close()
+
+    return retVal
+
+def set_peer_info(core_inst, peer, key, data):
+    '''
+        Update a peer for a key
+    '''
+
+    conn = sqlite3.connect(core_inst.peerDB, timeout=30)
+    c = conn.cursor()
+
+    command = (data, peer)
+
+    # TODO: validate key on whitelist
+    if key not in ('id', 'name', 'pubkey', 'forwardKey', 'dateSeen', 'trust'):
+        raise Exception("Got invalid database key when setting peer info")
+
+    c.execute('UPDATE peers SET ' + key + ' = ? WHERE id=?', command)
+    conn.commit()
+    conn.close()

onionr/dbcreator.py

@@ -1,5 +1,5 @@
 '''
-    Onionr - P2P Anonymous Data Storage & Sharing
+    Onionr - Private P2P Communication
 
     DBCreator, creates sqlite3 databases used by Onionr
 '''

onionr/dependencies/secrets.py

@@ -1,331 +0,0 @@
-"""Generate cryptographically strong pseudo-random numbers suitable for
-managing secrets such as account authentication, tokens, and similar.
-
-See PEP 506 for more information.
-https://www.python.org/dev/peps/pep-0506/
-
-
-A. HISTORY OF THE SOFTWARE
-==========================
-
-Python was created in the early 1990s by Guido van Rossum at Stichting
-Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands
-as a successor of a language called ABC.  Guido remains Python's
-principal author, although it includes many contributions from others.
-
-In 1995, Guido continued his work on Python at the Corporation for
-National Research Initiatives (CNRI, see http://www.cnri.reston.va.us)
-in Reston, Virginia where he released several versions of the
-software.
-
-In May 2000, Guido and the Python core development team moved to
-BeOpen.com to form the BeOpen PythonLabs team.  In October of the same
-year, the PythonLabs team moved to Digital Creations, which became
-Zope Corporation.  In 2001, the Python Software Foundation (PSF, see
-https://www.python.org/psf/) was formed, a non-profit organization
-created specifically to own Python-related Intellectual Property.
-Zope Corporation was a sponsoring member of the PSF.
-
-All Python releases are Open Source (see http://www.opensource.org for
-the Open Source Definition).  Historically, most, but not all, Python
-releases have also been GPL-compatible; the table below summarizes
-the various releases.
-
-    Release         Derived     Year        Owner       GPL-
-                    from                                compatible? (1)
-
-    0.9.0 thru 1.2              1991-1995   CWI         yes
-    1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes
-    1.6             1.5.2       2000        CNRI        no
-    2.0             1.6         2000        BeOpen.com  no
-    1.6.1           1.6         2001        CNRI        yes (2)
-    2.1             2.0+1.6.1   2001        PSF         no
-    2.0.1           2.0+1.6.1   2001        PSF         yes
-    2.1.1           2.1+2.0.1   2001        PSF         yes
-    2.1.2           2.1.1       2002        PSF         yes
-    2.1.3           2.1.2       2002        PSF         yes
-    2.2 and above   2.1.1       2001-now    PSF         yes
-
-Footnotes:
-
-(1) GPL-compatible doesn't mean that we're distributing Python under
-    the GPL.  All Python licenses, unlike the GPL, let you distribute
-    a modified version without making your changes open source.  The
-    GPL-compatible licenses make it possible to combine Python with
-    other software that is released under the GPL; the others don't.
-
-(2) According to Richard Stallman, 1.6.1 is not GPL-compatible,
-    because its license has a choice of law clause.  According to
-    CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1
-    is "not incompatible" with the GPL.
-
-Thanks to the many outside volunteers who have worked under Guido's
-direction to make these releases possible.
-
-
-B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON
-===============================================================
-
-PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
---------------------------------------------
-
-1. This LICENSE AGREEMENT is between the Python Software Foundation
-("PSF"), and the Individual or Organization ("Licensee") accessing and
-otherwise using this software ("Python") in source or binary form and
-its associated documentation.
-
-2. Subject to the terms and conditions of this License Agreement, PSF hereby
-grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce,
-analyze, test, perform and/or display publicly, prepare derivative works,
-distribute, and otherwise use Python alone or in any derivative version,
-provided, however, that PSF's License Agreement and PSF's notice of copyright,
-i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010,
-2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation; All
-Rights Reserved" are retained in Python alone or in any derivative version
-prepared by Licensee.
-
-3. In the event Licensee prepares a derivative work that is based on
-or incorporates Python or any part thereof, and wants to make
-the derivative work available to others as provided herein, then
-Licensee hereby agrees to include in any such work a brief summary of
-the changes made to Python.
-
-4. PSF is making Python available to Licensee on an "AS IS"
-basis.  PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
-IMPLIED.  BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND
-DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
-FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT
-INFRINGE ANY THIRD PARTY RIGHTS.
-
-5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
-FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
-A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON,
-OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
-
-6. This License Agreement will automatically terminate upon a material
-breach of its terms and conditions.
-
-7. Nothing in this License Agreement shall be deemed to create any
-relationship of agency, partnership, or joint venture between PSF and
-Licensee.  This License Agreement does not grant permission to use PSF
-trademarks or trade name in a trademark sense to endorse or promote
-products or services of Licensee, or any third party.
-
-8. By copying, installing or otherwise using Python, Licensee
-agrees to be bound by the terms and conditions of this License
-Agreement.
-
-
-BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0
--------------------------------------------
-
-BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1
-
-1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an
-office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the
-Individual or Organization ("Licensee") accessing and otherwise using
-this software in source or binary form and its associated
-documentation ("the Software").
-
-2. Subject to the terms and conditions of this BeOpen Python License
-Agreement, BeOpen hereby grants Licensee a non-exclusive,
-royalty-free, world-wide license to reproduce, analyze, test, perform
-and/or display publicly, prepare derivative works, distribute, and
-otherwise use the Software alone or in any derivative version,
-provided, however, that the BeOpen Python License is retained in the
-Software, alone or in any derivative version prepared by Licensee.
-
-3. BeOpen is making the Software available to Licensee on an "AS IS"
-basis.  BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
-IMPLIED.  BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND
-DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
-FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT
-INFRINGE ANY THIRD PARTY RIGHTS.
-
-4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE
-SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS
-AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY
-DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
-
-5. This License Agreement will automatically terminate upon a material
-breach of its terms and conditions.
-
-6. This License Agreement shall be governed by and interpreted in all
-respects by the law of the State of California, excluding conflict of
-law provisions.  Nothing in this License Agreement shall be deemed to
-create any relationship of agency, partnership, or joint venture
-between BeOpen and Licensee.  This License Agreement does not grant
-permission to use BeOpen trademarks or trade names in a trademark
-sense to endorse or promote products or services of Licensee, or any
-third party.  As an exception, the "BeOpen Python" logos available at
-http://www.pythonlabs.com/logos.html may be used according to the
-permissions granted on that web page.
-
-7. By copying, installing or otherwise using the software, Licensee
-agrees to be bound by the terms and conditions of this License
-Agreement.
-
-
-CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1
----------------------------------------
-
-1. This LICENSE AGREEMENT is between the Corporation for National
-Research Initiatives, having an office at 1895 Preston White Drive,
-Reston, VA 20191 ("CNRI"), and the Individual or Organization
-("Licensee") accessing and otherwise using Python 1.6.1 software in
-source or binary form and its associated documentation.
-
-2. Subject to the terms and conditions of this License Agreement, CNRI
-hereby grants Licensee a nonexclusive, royalty-free, world-wide
-license to reproduce, analyze, test, perform and/or display publicly,
-prepare derivative works, distribute, and otherwise use Python 1.6.1
-alone or in any derivative version, provided, however, that CNRI's
-License Agreement and CNRI's notice of copyright, i.e., "Copyright (c)
-1995-2001 Corporation for National Research Initiatives; All Rights
-Reserved" are retained in Python 1.6.1 alone or in any derivative
-version prepared by Licensee.  Alternately, in lieu of CNRI's License
-Agreement, Licensee may substitute the following text (omitting the
-quotes): "Python 1.6.1 is made available subject to the terms and
-conditions in CNRI's License Agreement.  This Agreement together with
-Python 1.6.1 may be located on the Internet using the following
-unique, persistent identifier (known as a handle): 1895.22/1013.  This
-Agreement may also be obtained from a proxy server on the Internet
-using the following URL: http://hdl.handle.net/1895.22/1013".
-
-3. In the event Licensee prepares a derivative work that is based on
-or incorporates Python 1.6.1 or any part thereof, and wants to make
-the derivative work available to others as provided herein, then
-Licensee hereby agrees to include in any such work a brief summary of
-the changes made to Python 1.6.1.
-
-4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS"
-basis.  CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
-IMPLIED.  BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND
-DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
-FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT
-INFRINGE ANY THIRD PARTY RIGHTS.
-
-5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
-1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
-A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1,
-OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
-
-6. This License Agreement will automatically terminate upon a material
-breach of its terms and conditions.
-
-7. This License Agreement shall be governed by the federal
-intellectual property law of the United States, including without
-limitation the federal copyright law, and, to the extent such
-U.S. federal law does not apply, by the law of the Commonwealth of
-Virginia, excluding Virginia's conflict of law provisions.
-Notwithstanding the foregoing, with regard to derivative works based
-on Python 1.6.1 that incorporate non-separable material that was
-previously distributed under the GNU General Public License (GPL), the
-law of the Commonwealth of Virginia shall govern this License
-Agreement only as to issues arising under or with respect to
-Paragraphs 4, 5, and 7 of this License Agreement.  Nothing in this
-License Agreement shall be deemed to create any relationship of
-agency, partnership, or joint venture between CNRI and Licensee.  This
-License Agreement does not grant permission to use CNRI trademarks or
-trade name in a trademark sense to endorse or promote products or
-services of Licensee, or any third party.
-
-8. By clicking on the "ACCEPT" button where indicated, or by copying,
-installing or otherwise using Python 1.6.1, Licensee agrees to be
-bound by the terms and conditions of this License Agreement.
-
-        ACCEPT
-
-
-CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2
---------------------------------------------------
-
-Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam,
-The Netherlands.  All rights reserved.
-
-Permission to use, copy, modify, and distribute this software and its
-documentation for any purpose and without fee is hereby granted,
-provided that the above copyright notice appear in all copies and that
-both that copyright notice and this permission notice appear in
-supporting documentation, and that the name of Stichting Mathematisch
-Centrum or CWI not be used in advertising or publicity pertaining to
-distribution of the software without specific, written prior
-permission.
-
-STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO
-THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
-FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE
-FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
-WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
-ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
-OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-
-
-"""
-
-__all__ = ['choice', 'randbelow', 'randbits', 'SystemRandom',
-           'token_bytes', 'token_hex', 'token_urlsafe',
-           'compare_digest',
-           ]
-
-
-import base64
-import binascii
-import os
-
-from hmac import compare_digest
-from random import SystemRandom
-
-_sysrand = SystemRandom()
-
-randbits = _sysrand.getrandbits
-choice = _sysrand.choice
-
-def randbelow(exclusive_upper_bound):
-    """Return a random int in the range [0, n)."""
-    if exclusive_upper_bound <= 0:
-        raise ValueError("Upper bound must be positive.")
-    return _sysrand._randbelow(exclusive_upper_bound)
-
-DEFAULT_ENTROPY = 32  # number of bytes to return by default
-
-def token_bytes(nbytes=None):
-    """Return a random byte string containing *nbytes* bytes.
-
-    If *nbytes* is ``None`` or not supplied, a reasonable
-    default is used.
-
-    >>> token_bytes(16)  #doctest:+SKIP
-    b'\\xebr\\x17D*t\\xae\\xd4\\xe3S\\xb6\\xe2\\xebP1\\x8b'
-
-    """
-    if nbytes is None:
-        nbytes = DEFAULT_ENTROPY
-    return os.urandom(nbytes)
-
-def token_hex(nbytes=None):
-    """Return a random text string, in hexadecimal.
-
-    The string has *nbytes* random bytes, each byte converted to two
-    hex digits.  If *nbytes* is ``None`` or not supplied, a reasonable
-    default is used.
-
-    >>> token_hex(16)  #doctest:+SKIP
-    'f9bf78b9a18ce6d46a0cd2b0b86df9da'
-
-    """
-    return binascii.hexlify(token_bytes(nbytes)).decode('ascii')
-
-def token_urlsafe(nbytes=None):
-    """Return a random URL-safe text string, in Base64 encoding.
-
-    The string has *nbytes* random bytes.  If *nbytes* is ``None``
-    or not supplied, a reasonable default is used.
-
-    >>> token_urlsafe(16)  #doctest:+SKIP
-    'Drmhze6EPcv0fN_81Bj-nA'
-
-    """
-    tok = token_bytes(nbytes)
-    return base64.urlsafe_b64encode(tok).rstrip(b'=').decode('ascii')
-

onionr/etc/README.md

@@ -0,0 +1,11 @@
+# etc
+
+Files that don't really fit anywhere else, but aren't used very frequently.
+
+## Files
+
+humanreadabletime.py: take integer seconds and return a human readable time string
+
+pgpwords.py: represent data using the pgp word list: https://en.wikipedia.org/wiki/PGP_word_list
+
+onionrvalues.py: spec values for onionr blocks and other things

onionr/etc/humanreadabletime.py

@@ -0,0 +1,38 @@
+'''
+    Onionr - Private P2P Communication
+
+    human_readable_time takes integer seconds and returns a human readable string
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+def human_readable_time(seconds):
+    build = ''
+
+    units = {
+        'year' : 31557600,
+        'month' : (31557600 / 12),
+        'day' : 86400,
+        'hour' : 3600,
+        'minute' : 60,
+        'second' : 1
+    }
+
+    for unit in units:
+        amnt_unit = int(seconds / units[unit])
+        if amnt_unit >= 1:
+            seconds -= amnt_unit * units[unit]
+            build += '%s %s' % (amnt_unit, unit) + ('s' if amnt_unit != 1 else '') + ' '
+
+    return build.strip()

onionr/etc/onionrvalues.py

@@ -1,5 +1,5 @@
 '''
-    Onionr - P2P Microblogging Platform & Social network
+    Onionr - Private P2P Communication
 
     This file defines values and requirements used by Onionr
 '''
@@ -17,8 +17,10 @@
     You should have received a copy of the GNU General Public License
     along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
-
+DENIABLE_PEER_ADDRESS = "OVPCZLOXD6DC5JHX4EQ3PSOGAZ3T24F75HQLIUZSDSMYPEOXCPFA===="
 class OnionrValues:
     def __init__(self):
         self.passwordLength = 20
-        self.blockMetadataLengths = {'meta': 1000, 'sig': 200, 'signer': 200, 'time': 10, 'powRandomToken': 1000, 'encryptType': 4, 'expire': 14} #TODO properly refine values to minimum needed
+        self.blockMetadataLengths = {'meta': 1000, 'sig': 200, 'signer': 200, 'time': 10, 'pow': 1000, 'encryptType': 4, 'expire': 14} #TODO properly refine values to minimum needed
+        self.default_expire = 2592000
+        self.announce_pow = 5

onionr/etc/pgpwords.py

@@ -3,6 +3,14 @@
 
 '''This file is adapted from https://github.com/thblt/pgp-words by github user 'thblt' ('Thibault Polge), GPL v3 license'''
 
+'''
+Changes made for Onionr by Kevin Froman in 2018-2019:
+Minor changes such as slight word adjustment, line breaks
+
+CLI commands/usage function removed
+hexify function added
+'''
+
 import os, re, sys, binascii
 
 _words = [

onionr/etc/powchoice.py

@@ -0,0 +1,27 @@
+'''
+    Onionr - Private P2P Communication
+
+    This file does determinations for what proof of work module should be used
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import platform
+def use_subprocess(core_inst):
+    use = True
+    if not core_inst.config.get('general.use_subprocess_pow_if_possible', True):
+        use = False
+    if 'Windows' == platform.system():
+        use = False
+    return use

onionr/httpapi/README.md

@@ -0,0 +1,13 @@
+# httpapi
+
+The httpapi contains collections of endpoints for the client and public API servers.
+
+## Files:
+
+configapi: manage onionr configuration from the client http api
+
+friendsapi: add, remove and list friends from the client http api
+
+miscpublicapi: misculanious onionr network interaction from the **public** httpapi, such as announcements, block fetching and uploading.
+
+profilesapi: work in progress in returning a profile page for an Onionr user

onionr/httpapi/__init__.py

@@ -1,5 +1,5 @@
 '''
-    Onionr - P2P Anonymous Storage Network
+    Onionr - Private P2P Communication
 
     This file registers plugin's flask blueprints for the client http server
 '''
@@ -19,11 +19,11 @@
 '''
 import onionrplugins
 
-def load_plugin_blueprints(flaskapp):
+def load_plugin_blueprints(flaskapp, blueprint='flask_blueprint'):
     '''Iterate enabled plugins and load any http endpoints they have'''
     for plugin in onionrplugins.get_enabled_plugins():
         plugin = onionrplugins.get_plugin(plugin)
         try:
-            flaskapp.register_blueprint(getattr(plugin, 'flask_blueprint'))
+            flaskapp.register_blueprint(getattr(plugin, blueprint))
         except AttributeError:
             pass

onionr/httpapi/apiutils/__init__.py

@@ -0,0 +1,3 @@
+from . import shutdown, setbindip, getblockdata
+
+GetBlockData = getblockdata.GetBlockData

onionr/httpapi/apiutils/getblockdata.py

@@ -0,0 +1,42 @@
+import json
+import core, onionrblockapi
+from onionrutils import bytesconverter, stringvalidators
+class GetBlockData:
+    def __init__(self, client_api_inst=None):
+        if client_api_inst is None:
+            self.client_api_inst = None
+            self.c = core.Core()
+        elif isinstance(client_api_inst, core.Core):
+            self.client_api_inst = None
+            self.c = client_api_inst
+        else:
+            self.client_api_Inst = client_api_inst
+            self.c = core.Core()
+    
+    def get_block_data(self, bHash, decrypt=False, raw=False, headerOnly=False):
+        assert stringvalidators.validate_hash(bHash)
+        bl = onionrblockapi.Block(bHash, core=self.c)
+        if decrypt:
+            bl.decrypt()
+            if bl.isEncrypted and not bl.decrypted:
+                raise ValueError
+
+        if not raw:
+            if not headerOnly:
+                retData = {'meta':bl.bheader, 'metadata': bl.bmetadata, 'content': bl.bcontent}
+                for x in list(retData.keys()):
+                    try:
+                        retData[x] = retData[x].decode()
+                    except AttributeError:
+                        pass
+            else:
+                validSig = False
+                signer = bytesconverter.bytes_to_str(bl.signer)
+                if bl.isSigned() and stringvalidators.validate_pub_key(signer) and bl.isSigner(signer):
+                    validSig = True                    
+                bl.bheader['validSig'] = validSig
+                bl.bheader['meta'] = ''
+                retData = {'meta': bl.bheader, 'metadata': bl.bmetadata}
+            return json.dumps(retData)
+        else:
+            return bl.raw

onionr/httpapi/apiutils/setbindip.py

@@ -0,0 +1,25 @@
+import random, socket
+import config, logger
+def set_bind_IP(filePath='', core_inst=None):
+    '''Set a random localhost IP to a specified file (intended for private or public API localhost IPs)'''
+    if not core_inst is None:
+        config = core_inst.config
+
+    if config.get('general.random_bind_ip', True):
+        hostOctets = [str(127), str(random.randint(0x02, 0xFF)), str(random.randint(0x02, 0xFF)), str(random.randint(0x02, 0xFF))]
+        data = '.'.join(hostOctets)
+        # Try to bind IP. Some platforms like Mac block non normal 127.x.x.x
+        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        try:
+            s.bind((data, 0))
+        except OSError:
+            # if mac/non-bindable, show warning and default to 127.0.0.1
+            logger.warn('Your platform appears to not support random local host addresses 127.x.x.x. Falling back to 127.0.0.1.')
+            data = '127.0.0.1'
+        s.close()
+    else:
+        data = '127.0.0.1'
+    if filePath != '':
+        with open(filePath, 'w') as bindFile:
+            bindFile.write(data)
+    return data

onionr/httpapi/apiutils/shutdown.py

@@ -0,0 +1,38 @@
+'''
+    Onionr - Private P2P Communication
+
+    Shutdown the node either hard or cleanly
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+from flask import Blueprint, Response
+import core, onionrblockapi, onionrexceptions
+from onionrutils import stringvalidators
+
+shutdown_bp = Blueprint('shutdown', __name__)
+
+def shutdown(client_api_inst):
+    try:
+        client_api_inst.publicAPI.httpServer.stop()
+        client_api_inst.httpServer.stop()
+    except AttributeError:
+        pass
+    return Response("bye")
+
+@shutdown_bp.route('/shutdownclean')
+def shutdown_clean():
+    # good for calling from other clients
+    core.Core().daemonQueueAdd('shutdown')
+    return Response("bye")

onionr/httpapi/configapi/__init__.py

@@ -0,0 +1,62 @@
+'''
+    Onionr - Private P2P Communication
+
+    This file handles configuration setting and getting from the HTTP API
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import json
+from flask import Blueprint, request, Response, abort
+import config, onionrutils
+config.reload()
+
+config_BP = Blueprint('config_BP', __name__)
+
+@config_BP.route('/config/get')
+def get_all_config():
+    '''Simply return all configuration as JSON string'''
+    return Response(json.dumps(config.get_config(), indent=4, sort_keys=True))
+
+@config_BP.route('/config/get/<key>')
+def get_by_key(key):
+    '''Return a config setting by key'''
+    return Response(json.dumps(config.get(key)))
+
+@config_BP.route('/config/setall', methods=['POST'])
+def set_all_config():
+    '''Overwrite existing JSON config with new JSON string'''
+    try:
+        new_config = request.get_json(force=True)
+    except json.JSONDecodeError:
+        abort(400)
+    else:
+        config.set_config(new_config)
+        config.save()
+        return Response('success')
+
+@config_BP.route('/config/set/<key>', methods=['POST'])
+def set_by_key(key):
+    '''Overwrite/set only 1 config key'''
+    '''
+    {
+        'data': data
+    }
+    '''
+    try:
+        data = json.loads(onionrutils.OnionrUtils.bytesToStr(request.data))['data']
+    except (json.JSONDecodeError, KeyError):
+        abort(400)
+    config.set(key, data, True)
+    return Response('success')

onionr/httpapi/fdsafehandler.py

@@ -0,0 +1,21 @@
+from gevent.pywsgi import WSGIServer, WSGIHandler
+from gevent import Timeout
+class FDSafeHandler(WSGIHandler):
+    '''Our WSGI handler. Doesn't do much non-default except timeouts'''
+    def handle(self):
+        self.timeout = Timeout(120, Exception)
+        self.timeout.start()
+        try:
+            WSGIHandler.handle(self)
+        except Exception:
+            self.handle_error()
+        finally:
+            self.timeout.close()
+
+    def handle_error(self):
+        if v is self.timeout:
+            self.result = [b"Timeout"]
+            self.start_response("200 OK", [])
+            self.process_result()
+        else:
+            WSGIHandler.handle_error(self) 

onionr/httpapi/friendsapi/__init__.py

@@ -1,5 +1,5 @@
 '''
-    Onionr - P2P Anonymous Storage Network
+    Onionr - Private P2P Communication
 
     This file creates http endpoints for friend management
 '''
@@ -26,7 +26,8 @@ friends = Blueprint('friends', __name__)
 @friends.route('/friends/list')
 def list_friends():
     pubkey_list = {}
-    friend_list = contactmanager.ContactManager.list_friends(core.Core())
+    c = core.Core()
+    friend_list = contactmanager.ContactManager.list_friends(c)
     for friend in friend_list:
         pubkey_list[friend.publicKey] = {'name': friend.get_info('name')}
     return json.dumps(pubkey_list)

onionr/httpapi/insertblock.py

@@ -0,0 +1,63 @@
+'''
+    Onionr - Private P2P Communication
+
+    Create blocks with the client api server
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import json, threading
+from flask import Blueprint, Response, request
+import core
+
+ib = Blueprint('insertblock', __name__)
+
+@ib.route('/insertblock', methods=['POST'])
+def client_api_insert_block():
+    c = core.Core()
+    encrypt = False
+    bData = request.get_json(force=True)
+    message = bData['message']
+
+    # Detect if message (block body) is not specified
+    if type(message) is None:
+        return 'failure', 406
+
+    subject = 'temp'
+    encryptType = ''
+    sign = True
+    meta = {}
+    to = ''
+    try:
+        if bData['encrypt']:
+            to = bData['to']
+            encrypt = True
+            encryptType = 'asym'
+    except KeyError:
+        pass
+    try:
+        if not bData['sign']:
+            sign = False
+    except KeyError:
+        pass
+    try:
+        bType = bData['type']
+    except KeyError:
+        bType = 'bin'
+    try:
+        meta = json.loads(bData['meta'])
+    except KeyError:
+        pass
+    threading.Thread(target=c.insertBlock, args=(message,), kwargs={'header': bType, 'encryptType': encryptType, 'sign':sign, 'asymPeer': to, 'meta': meta}).start()
+    return Response('success')

onionr/httpapi/miscclientapi/__init__.py

@@ -0,0 +1 @@
+from . import getblocks, staticfiles, endpoints

onionr/httpapi/miscclientapi/endpoints.py

@@ -0,0 +1,112 @@
+'''
+    Onionr - Private P2P Communication
+
+    Misc client API endpoints too small to need their own file and that need access to the client api inst
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+from flask import Response, Blueprint, request, send_from_directory, abort
+from httpapi import apiutils
+class PrivateEndpoints:
+    def __init__(self, client_api):
+        private_endpoints_bp = Blueprint('privateendpoints', __name__)
+        self.private_endpoints_bp = private_endpoints_bp
+        config = client_api._core.config
+
+        @private_endpoints_bp.route('/serviceactive/<pubkey>')
+        def serviceActive(pubkey):
+            try:
+                if pubkey in client_api._core.onionrInst.communicatorInst.active_services:
+                    return Response('true')
+            except AttributeError as e:
+                pass
+            return Response('false')
+
+        @private_endpoints_bp.route('/www/<path:path>', endpoint='www')
+        def wwwPublic(path):
+            if not config.get("www.private.run", True):
+                abort(403)
+            return send_from_directory(config.get('www.private.path', 'static-data/www/private/'), path)
+
+        @private_endpoints_bp.route('/hitcount')
+        def get_hit_count():
+            return Response(str(client_api.publicAPI.hitCount))
+
+        @private_endpoints_bp.route('/queueResponseAdd/<name>', methods=['post'])
+        def queueResponseAdd(name):
+            # Responses from the daemon. TODO: change to direct var access instead of http endpoint
+            client_api.queueResponse[name] = request.form['data']
+            return Response('success')
+        
+        @private_endpoints_bp.route('/queueResponse/<name>')
+        def queueResponse(name):
+            # Fetch a daemon queue response
+            resp = 'failure'
+            try:
+                resp = client_api.queueResponse[name]
+            except KeyError:
+                pass
+            else:
+                del client_api.queueResponse[name]
+            if resp == 'failure':
+                return resp, 404
+            else:
+                return resp
+            
+        @private_endpoints_bp.route('/ping')
+        def ping():
+            # Used to check if client api is working
+            return Response("pong!")
+
+        @private_endpoints_bp.route('/lastconnect')
+        def lastConnect():
+            return Response(str(client_api.publicAPI.lastRequest))
+
+        @private_endpoints_bp.route('/waitforshare/<name>', methods=['post'])
+        def waitforshare(name):
+            '''Used to prevent the **public** api from sharing blocks we just created'''
+            assert name.isalnum()
+            if name in client_api.publicAPI.hideBlocks:
+                client_api.publicAPI.hideBlocks.remove(name)
+                return Response("removed")
+            else:
+                client_api.publicAPI.hideBlocks.append(name)
+                return Response("added")
+
+        @private_endpoints_bp.route('/shutdown')
+        def shutdown():
+            return apiutils.shutdown.shutdown(client_api)
+        
+        @private_endpoints_bp.route('/getstats')
+        def getStats():
+            # returns node stats
+            #return Response("disabled")
+            while True:
+                try:    
+                    return Response(client_api._core.serializer.getStats())
+                except AttributeError:
+                    pass
+        
+        @private_endpoints_bp.route('/getuptime')
+        def showUptime():
+            return Response(str(client_api.getUptime()))
+        
+        @private_endpoints_bp.route('/getActivePubkey')
+        def getActivePubkey():
+            return Response(client_api._core._crypto.pubKey)
+
+        @private_endpoints_bp.route('/getHumanReadable/<name>')
+        def getHumanReadable(name):
+            return Response(mnemonickeys.get_human_readable_ID(name))

onionr/httpapi/miscclientapi/getblocks.py

@@ -0,0 +1,66 @@
+'''
+    Onionr - Private P2P Communication
+
+    Create blocks with the client api server
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+from flask import Blueprint, Response, abort
+import core, onionrblockapi
+from httpapi import apiutils
+from onionrutils import stringvalidators
+
+c = core.Core()
+
+client_get_block = apiutils.GetBlockData(c)
+
+client_get_blocks = Blueprint('miscclient', __name__)
+
+@client_get_blocks.route('/getblocksbytype/<name>')
+def getBlocksByType(name):
+    blocks = c.getBlocksByType(name)
+    return Response(','.join(blocks))
+
+@client_get_blocks.route('/getblockbody/<name>')
+def getBlockBodyData(name):
+    resp = ''
+    if stringvalidators.validate_hash(name):
+        try:
+            resp = onionrblockapi.Block(name, decrypt=True, core=c).bcontent
+        except TypeError:
+            pass
+    else:
+        abort(404)
+    return Response(resp)
+
+@client_get_blocks.route('/getblockdata/<name>')
+def getData(name):
+    resp = ""
+    if stringvalidators.validate_hash(name):
+        if name in c.getBlockList():
+            try:
+                resp = client_get_block.get_block_data(name, decrypt=True)
+            except ValueError:
+                pass
+        else:
+            abort(404)
+    else:
+        abort(404)
+    return Response(resp)
+
+@client_get_blocks.route('/getblockheader/<name>')
+def getBlockHeader(name):
+    resp = client_get_block.get_block_data(name, decrypt=True, headerOnly=True)
+    return Response(resp)

onionr/httpapi/miscclientapi/staticfiles.py

@@ -0,0 +1,70 @@
+'''
+    Onionr - Private P2P Communication
+
+    Register static file routes
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import os
+from flask import Blueprint, send_from_directory
+
+static_files_bp = Blueprint('staticfiles', __name__)
+
+root = os.getcwd() + '/static-data/www/' # should be set to onionr install directory from onionr startup
+
+@static_files_bp.route('/board/', endpoint='board')
+def loadBoard():
+    return send_from_directory(root + 'board/', "index.html")
+
+@static_files_bp.route('/mail/<path:path>', endpoint='mail')
+def loadMail(path):
+    return send_from_directory(root + 'mail/', path)
+
+@static_files_bp.route('/mail/', endpoint='mailindex')
+def loadMailIndex():
+    return send_from_directory(root + 'mail/', 'index.html')
+
+@static_files_bp.route('/friends/<path:path>', endpoint='friends')
+def loadContacts(path):
+    return send_from_directory(root + 'friends/', path)
+
+@static_files_bp.route('/friends/', endpoint='friendsindex')
+def loadContacts():
+    return send_from_directory(root + 'friends/', 'index.html')
+
+@static_files_bp.route('/profiles/<path:path>', endpoint='profiles')
+def loadContacts(path):
+    return send_from_directory(root + 'profiles/', path)
+
+@static_files_bp.route('/profiles/', endpoint='profilesindex')
+def loadContacts():
+    return send_from_directory(root + 'profiles/', 'index.html')
+
+@static_files_bp.route('/board/<path:path>', endpoint='boardContent')
+def boardContent(path):
+    return send_from_directory(root + 'board/', path)
+
+@static_files_bp.route('/shared/<path:path>', endpoint='sharedContent')
+def sharedContent(path):
+    return send_from_directory(root + 'shared/', path)
+
+@static_files_bp.route('/', endpoint='onionrhome')
+def hello():
+    # ui home
+    return send_from_directory(root + 'private/', 'index.html')
+
+@static_files_bp.route('/private/<path:path>', endpoint='homedata')
+def homedata(path):
+    return send_from_directory(root + 'private/', path)

onionr/httpapi/miscpublicapi/__init__.py

@@ -0,0 +1,6 @@
+from . import announce, upload, getblocks, endpoints
+
+announce = announce.handle_announce # endpoint handler for accepting peer announcements
+upload = upload.accept_upload # endpoint handler for accepting public uploads
+public_block_list = getblocks.get_public_block_list # endpoint handler for getting block lists
+public_get_block_data = getblocks.get_block_data # endpoint handler for responding to peers requests for block data

onionr/httpapi/miscpublicapi/announce.py

@@ -0,0 +1,65 @@
+'''
+    Onionr - Private P2P Communication
+
+    Handle announcements to the public API server
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import base64
+from flask import Response
+import logger
+from etc import onionrvalues
+from onionrutils import stringvalidators, bytesconverter
+
+def handle_announce(clientAPI, request):
+    '''
+    accept announcement posts, validating POW
+    clientAPI should be an instance of the clientAPI server running, request is a instance of a flask request
+    '''
+    resp = 'failure'
+    powHash = ''
+    randomData = ''
+    newNode = ''
+    ourAdder = clientAPI._core.hsAddress.encode()
+    try:
+        newNode = request.form['node'].encode()
+    except KeyError:
+        logger.warn('No node specified for upload')
+        pass
+    else:
+        try:
+            randomData = request.form['random']
+            randomData = base64.b64decode(randomData)
+        except KeyError:
+            logger.warn('No random data specified for upload')
+        else:
+            nodes = newNode + clientAPI._core.hsAddress.encode()
+            nodes = clientAPI._core._crypto.blake2bHash(nodes)
+            powHash = clientAPI._core._crypto.blake2bHash(randomData + nodes)
+            try:
+                powHash = powHash.decode()
+            except AttributeError:
+                pass
+            if powHash.startswith('0' * onionrvalues.OnionrValues().announce_pow):
+                newNode = bytesconverter.bytes_to_str(newNode)
+                if stringvalidators.validate_transport(newNode) and not newNode in clientAPI._core.onionrInst.communicatorInst.newPeers:
+                    clientAPI._core.onionrInst.communicatorInst.newPeers.append(newNode)
+                    resp = 'Success'
+            else:
+                logger.warn(newNode.decode() + ' failed to meet POW: ' + powHash)
+    resp = Response(resp)
+    if resp == 'failure':
+        return resp, 406
+    return resp

onionr/httpapi/miscpublicapi/endpoints.py

@@ -0,0 +1,80 @@
+'''
+    Onionr - Private P2P Communication
+
+    Misc public API endpoints too small to need their own file and that need access to the public api inst
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+from flask import Response, Blueprint, request, send_from_directory, abort
+from . import getblocks, upload, announce
+class PublicEndpoints:
+    def __init__(self, public_api):
+        client_API = public_api.clientAPI
+        config = client_API._core.config
+
+        public_endpoints_bp = Blueprint('publicendpoints', __name__)
+        self.public_endpoints_bp = public_endpoints_bp
+
+        @public_endpoints_bp.route('/')
+        def banner():
+            # Display a bit of information to people who visit a node address in their browser
+            try:
+                with open('static-data/index.html', 'r') as html:
+                    resp = Response(html.read(), mimetype='text/html')
+            except FileNotFoundError:
+                resp = Response("")
+            return resp
+
+        @public_endpoints_bp.route('/getblocklist')
+        def get_block_list():
+            '''Get a list of blocks, optionally filtered by epoch time stamp, excluding those hidden'''
+            return getblocks.get_public_block_list(client_API, public_api, request)
+
+        @public_endpoints_bp.route('/getdata/<name>')
+        def get_block_data(name):
+            # Share data for a block if we have it and it isn't hidden
+            return getblocks.get_block_data(client_API, public_api, name)
+
+        @public_endpoints_bp.route('/www/<path:path>')
+        def www_public(path):
+            # A way to share files directly over your .onion
+            if not config.get("www.public.run", True):
+                abort(403)
+            return send_from_directory(config.get('www.public.path', 'static-data/www/public/'), path)
+
+        @public_endpoints_bp.route('/ping')
+        def ping():
+            # Endpoint to test if nodes are up
+            return Response("pong!")
+
+        @public_endpoints_bp.route('/pex')
+        def peer_exchange():
+            response = ','.join(client_API._core.listAdders(recent=3600))
+            if len(response) == 0:
+                response = ''
+            return Response(response)
+
+        @public_endpoints_bp.route('/announce', methods=['post'])
+        def accept_announce():
+            '''Accept announcements with pow token to prevent spam'''
+            resp = announce.handle_announce(client_API, request)
+            return resp
+
+        @public_endpoints_bp.route('/upload', methods=['post'])
+        def upload_endpoint():
+            '''Accept file uploads. In the future this will be done more often than on creation 
+            to speed up block sync
+            '''
+            return upload.accept_upload(client_API, request)

onionr/httpapi/miscpublicapi/getblocks.py

@@ -0,0 +1,52 @@
+'''
+    Onionr - Private P2P Communication
+
+    Public endpoints to get block data and lists
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+from flask import Response, abort
+import config
+from onionrutils import bytesconverter, stringvalidators
+
+def get_public_block_list(clientAPI, publicAPI, request):
+    # Provide a list of our blocks, with a date offset
+    dateAdjust = request.args.get('date')
+    bList = clientAPI._core.getBlockList(dateRec=dateAdjust)
+    if clientAPI._core.config.get('general.hide_created_blocks', True):
+        for b in publicAPI.hideBlocks:
+            if b in bList:
+                # Don't share blocks we created if they haven't been *uploaded* yet, makes it harder to find who created a block
+                bList.remove(b)
+    return Response('\n'.join(bList))
+
+def get_block_data(clientAPI, publicAPI, data):
+    '''data is the block hash in hex'''
+    resp = ''
+    if stringvalidators.validate_hash(data):
+        if not clientAPI._core.config.get('general.hide_created_blocks', True) or data not in publicAPI.hideBlocks:
+            if data in clientAPI._core.getBlockList():
+                block = clientAPI.getBlockData(data, raw=True)
+                try:
+                    block = block.encode() # Encode in case data is binary
+                except AttributeError:
+                    abort(404)
+                block = bytesconverter.str_to_bytes(block)
+                resp = block
+    if len(resp) == 0:
+        abort(404)
+        resp = ""
+    # Has to be octet stream, otherwise binary data fails hash check
+    return Response(resp, mimetype='application/octet-stream')

onionr/httpapi/miscpublicapi/upload.py

@@ -0,0 +1,43 @@
+'''
+    Onionr - Private P2P Communication
+
+    Accept block uploads to the public API server
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import sys
+from flask import Response, abort
+import blockimporter, onionrexceptions, logger
+def accept_upload(clientAPI, request):
+    resp = 'failure'
+    try:
+        data = request.form['block']
+    except KeyError:
+        logger.warn('No block specified for upload')
+        pass
+    else:
+        if sys.getsizeof(data) < 100000000:
+            try:
+                if blockimporter.importBlockFromData(data, clientAPI._core):
+                    resp = 'success'
+                else:
+                    logger.warn('Error encountered importing uploaded block')
+            except onionrexceptions.BlacklistedBlock:
+                logger.debug('uploaded block is blacklisted')
+                pass
+    if resp == 'failure':
+        abort(400)
+    resp = Response(resp)
+    return resp

onionr/httpapi/onionrsitesapi/__init__.py

@@ -0,0 +1,44 @@
+'''
+    Onionr - Private P2P Communication
+
+    view and interact with onionr sites
+'''
+'''
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+'''
+import base64
+from flask import Blueprint, Response, request, abort
+import core, onionrblockapi, onionrexceptions
+from onionrutils import stringvalidators
+
+site_api = Blueprint('siteapi', __name__)
+
+@site_api.route('/site/<name>', endpoint='site')
+def site(name):
+    bHash = name
+    resp = 'Not Found'
+    if stringvalidators.validate_hash(bHash):
+        try:
+            resp = onionrblockapi.Block(bHash).bcontent
+        except onionrexceptions.NoDataAvailable:
+            abort(404)
+        except TypeError:
+            pass
+        try:
+            resp = base64.b64decode(resp)
+        except:
+            pass
+    if resp == 'Not Found' or not resp:
+        abort(404)
+    return Response(resp)

onionr/httpapi/profilesapi/__init__.py

@@ -1,7 +1,7 @@
 '''
-    Onionr - P2P Anonymous Storage Network
+    Onionr - Private P2P Communication
 
-    This file creates http endpoints for friend management
+    This file creates http endpoints for user profile pages
 '''
 '''
     This program is free software: you can redistribute it and/or modify
@@ -19,13 +19,10 @@
 '''
 import core
 from flask import Blueprint, Response, request, abort
+from . import profiles
 
-simplecache = Blueprint('simplecache', __name__)
+profile_BP = Blueprint('profile_BP', __name__)
 
-@simplecache.route('/get/<key>')
+@profile_BP.route('/profile/get/<pubkey>', endpoint='profiles')
-def get_key(key):
+def get_profile_page(pubkey):
-    return
+    return Response(pubkey)
-
-@simplecache.route('/set/<key>', methods=['POST'])
-def set_key(key):
-    return

onionr/httpapi/profilesapi/profiles.py

@@ -0,0 +1,2 @@
+def get_latest_user_profile(pubkey):
+    return ''