kev/Onionr
1
0
Fork 0
Code Issues 24 Pull Requests Projects Releases 2 Wiki Activity

Merge branch 'master' of ssh://gitlab.com/beardog/onionr

Browse Source
This commit is contained in:
Arinerron 2019-07-22 16:41:47 -07:00
commit 2b17cd13f4
No known key found for this signature in database
GPG Key ID: 99383627861C62F0
237 changed files with 7816 additions and 5341 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.github/FUNDING.yml vendored Executable file
View File

@ -0,0 +1,4 @@
# These are supported funding model platforms
ko_fi: beardogkf

3
.gitignore vendored
View File

@ -18,6 +18,9 @@ core
venv/*
onionr/fs*
*.dll
*.exe
# log files
output.log
*.log

6
CODE_OF_CONDUCT.md Executable file → Normal file
View File

@ -40,7 +40,7 @@ Project maintainers who do not follow or enforce the Code of Conduct in good fai
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://contributor-covenant.org/version/1/4][version]
[homepage]: http://contributor-covenant.org
[version]: http://contributor-covenant.org/version/1/4/
[homepage]: https://contributor-covenant.org
[version]: https://contributor-covenant.org/version/1/4/

10
CONTRIBUTING.md Executable file → Normal file
View File

@ -1,12 +1,12 @@
# Contributing to Onionr
One of the great things about open source projects is that they allow for many people to contribute to the project.
One of the great things about open source projects is that they allow for anyone to contribute to the project.
This file serves to provide guidelines on how to successfully contribute to Onionr.
## Code of Conduct
See our [Code of Conduct](https://github.com/beardog108/onionr/blob/master/CODE_OF_CONDUCT.md)
Contributors in project-related spaces/contexts are expected to follow the [Code of Conduct](https://github.com/beardog108/onionr/blob/master/CODE_OF_CONDUCT.md)
## Reporting Bugs
@ -23,11 +23,13 @@ Please provide the following information when reporting a bug:
If a bug is a security issue, please contact us privately.
And most importantly, please be patient. Onionr is an open source project done by volunteers.
And most importantly, please be patient. Onionr is a free open source project maintained by volunteers in their free time.
## Asking Questions
If you need help with Onionr, you can contact the devs (be polite and remember this is a volunteer-driven non-profit project).
If you need help with Onionr, you can contact the devs (be polite and remember this is a volunteer-driven project).
Do your best to use good english.
## Contributing Code

0
Dockerfile Executable file → Normal file
View File

0
ISSUE_TEMPLATE.md Executable file → Normal file
View File

3
LICENSE.txt Executable file → Normal file
View File

@ -1,6 +1,3 @@
The Onionr logo was created by [Anhar Ismail](https://github.com/anharismail) under the [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007

82
README.md Executable file → Normal file
View File

@ -5,28 +5,30 @@
</p>
<p align="center">
Anonymous P2P storage network 🕵️
Private P2P Communication Network 🕵️
</p>
(***pre-alpha & experimental, not well tested or easy to use yet***)
[![Open Source Love](https://badges.frapsoft.com/os/v3/open-source.png?v=103)](https://github.com/ellerbrock/open-source-badges/)
<img src='https://gitlab.com/beardog/Onionr/badges/master/build.svg'>
<img src='https://gitlab.com/beardog/Onionr/badges/master/build.svg'> - [Onionr.net](https://onionr.net/) - [.onion](http://onionr.onionkvc5ibm37bmxwr56bdxcdnb6w3wm4bdghh5qo6f6za7gn7styid.onion/)
<hr>
**The main repository for this software is at https://gitlab.com/beardog/Onionr/**
# Summary
# About
Onionr is a decentralized, peer-to-peer data storage network, designed to be anonymous and resistant to (meta)data analysis and spam/disruption.
Onionr is a decentralized, peer-to-peer communication network, designed to be anonymous and resistant to (meta)data analysis, spam, and corruption.
Onionr stores data in independent packages referred to as 'blocks'. The blocks are synced to all other nodes in the network. Blocks and user IDs cannot be easily proven to have been created by particular nodes (only inferred). Even if there is enough evidence to believe a particular node created a block, nodes still operate behind Tor or I2P and as such are not trivially known to be at a particular IP address.
Onionr stores data in independent packages referred to as 'blocks'. The blocks are synced to all other nodes in the network. Blocks and user IDs cannot be easily proven to have been created by a particular user. Even if there is enough evidence to believe that a specific user created a block, nodes still operate behind Tor or I2P and as such cannot be trivially unmasked.
Users are identified by ed25519 public keys, which can be used to sign blocks or send encrypted data.
Users are identified by ed25519/curve25519 public keys, which can be used to sign blocks or send encrypted data.
Onionr can be used for mail, as a social network, instant messenger, file sharing software, or for encrypted group discussion.
The whitepaper (subject to change prior to alpha release) is available [here](docs/whitepaper.md).
![Tor stinks slide image](docs/tor-stinks-02.png)
## Main Features
@ -34,17 +36,33 @@ Onionr can be used for mail, as a social network, instant messenger, file sharin
* [X] 🌐 Fully p2p/decentralized, no trackers or other single points of failure
* [X] 🔒 End to end encryption of user data
* [X] 📢 Optional non-encrypted blocks, useful for blog posts or public file sharing
* [X] 💻 Easy API system for integration to websites
* [X] 💻 Easy HTTP API for integration to websites
* [X] 🕵️ Metadata analysis resistance and anonymity
* [X] 📡 Transport agnosticism (no internet required)
## Software Suite
Onionr ships with various application plugins ready for use out of the box:
Currently usable:
* Mail
* Public anonymous chat
* Simple webpage hosting - Will be greatly extended
* File sharing (Work in progress)
Not yet usable:
* Instant messaging
* Forum/BBS
**Onionr API and functionality is subject to non-backwards compatible change during pre-alpha development**
# Screenshots
<img alt='Node statistics page screenshot' src='docs/onionr-1.png' width=600>
Node statistics
Home screen
<img alt='Friend/contact manager screenshot' src='docs/onionr-2.png' width=600>
@ -52,22 +70,31 @@ Friend/contact manager
<img alt='Encrypted, metadata-masking mail application screenshot' src='docs/onionr-3.png' width=600>
Encrypted, metadata-masking mail application.
Encrypted, metadata-masking mail application. One of the first distributed mail systems to have basic forward secrecy.
# Documentation
More docs coming soon.
* [Block specification](docs/specs/block-spec.md)
* [HTTP API](docs/http-api.md)
# Install and Run on Linux
The following applies to Ubuntu Bionic. Other distros may have different package or command names.
* Have python3.6+, python3-pip, Tor (daemon, not browser) installed (python3-dev recommended)
`$ sudo apt install python3-pip python3-dev tor`
* Have python3.6+, python3-pip, Tor (daemon, not browser) installed. python3-dev is recommended.
* Clone the git repo: `$ git clone https://gitlab.com/beardog/onionr`
* cd into install direction: `$ cd onionr/`
* Install the Python dependencies ([virtualenv strongly recommended](https://virtualenv.pypa.io/en/stable/userguide/)): `$ pip3 install --require-hashes -r requirements.txt`
(--require-hashes is intended to prevent exploitation via compromise of Pypi/CA certificates)
(--require-hashes is intended to prevent exploitation via compromise of PyPi/CA certificates)
## Help out
# Help out
Everyone is welcome to help out. Help is wanted for the following:
Everyone is welcome to contribute. Help is wanted for the following:
* Development (Get in touch first)
* Creation of a shared lib for use from other languages and faster proof-of-work
@ -75,26 +102,39 @@ Everyone is welcome to help out. Help is wanted for the following:
* Windows and Mac support (already partially supported, testers needed)
* General bug fixes and development of new features
* Testing
* Translations/localizations
* UI/UX design
* Running stable nodes
* Security review/audit
* Automatic I2P setup
Contribute money:
## Contribute money:
Donating at least $5 gets you cool Onionr stickers. Get in touch if you want them.
Bitcoin: [1onion55FXzm6h8KQw3zFw2igpHcV7LPq](bitcoin:1onion55FXzm6h8KQw3zFw2igpHcV7LPq) (Contact us for privacy coins like Monero)
Bitcoin: [1onion55FXzm6h8KQw3zFw2igpHcV7LPq](bitcoin:1onion55FXzm6h8KQw3zFw2igpHcV7LPq) (Contact us for a unique address or for other coins)
Monero: 4B5BA24d1P3R5aWEpkGY5TP7buJJcn2aSGBVRQCHhpiahxeB4aWsu15XwmuTjC6VF62NApZeJGTS248RMVECP8aW73Uj2ax
USD (Card/Paypal): [Ko-Fi](https://www.ko-fi.com/beardogkf)
Note: probably not tax deductible
## Contact
# Contact
beardog [ at ] mailbox.org
Email: beardog [ at ] mailbox.org
## Disclaimer
Onionr Mail: TRH763JURNY47QPBTTQ4LLPYCYQK6Q5YA33R6GANKZK5C5DKCIGQ
## Disclaimers and legal
No matter how good Onionr and other software gets, there will always be ways for clever or well-funded adversaries to break your security.
*Do not rely on Onionr or any other software to hold up if your life or liberty are at stake.*
### Licenses and Branding
Onionr is published under the GNU GPL v3 license, except for the logo.
The Tor Project and I2P developers do not own, create, or endorse this project, and are not otherwise involved.
@ -102,8 +142,10 @@ Tor is a trademark for the Tor Project. We do not own it.
The 'open source badge' is by Maik Ellerbrock and is licensed under a Creative Commons Attribution 4.0 International License.
## Logo
## Onionr Logo
The Onionr logo was created by [Anhar Ismail](https://github.com/anharismail) under the [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).
If you modify and redistribute our code ("forking"), please use a different logo and project name to avoid confusion. Please do not use our logo in a way that makes it seem like we endorse you without permission.
If you modify and redistribute our code ("forking"), please use a different logo and project name to avoid confusion. Please do not use the project name or logo in a way that makes it seem like we endorse you without our permission.
![node web illustration](docs/onionr-web.png)

39
SECURITY.md Normal file
View File

@ -0,0 +1,39 @@
# Security Policy
We welcome responsible and constructive security review.
# Scope
The Onionr software and any nodes you control are within scope.
Avoid social engineering, volume-based denial of service and disrupting or harming the Onionr network. Do not attempt to exploit any machines/servers you do not own or otherwise have permission to do so.
The following exploits are of particular interest:
* Arbitrary code execution
* API authentication bypass (such as accessing local API from public interface)
* Deanonymization:
* Easily associating public keys with server addresses
* Discovering true server IPs when behind Tor/I2P (aside from Tor/i2p-level attacks)
* Easily discovering which nodes are the block creator
* XSS, CSRF, clickjacking, DNS rebinding
* Timing attacks against the local http server ([see blog post](https://www.chaoswebs.net/blog/timebleed-breaking-privacy-with-a-simple-timing-attack.html))
* Discovering direct connection servers as a non participant.
* Cryptography/protocol issues
* Denying nodes access to the network by segmenting them out with Sybil nodes
We do not consider non-network based same-machine attacks to be very significant, but we are still willing to listen.
# Rewards
Onionr is a student-owned hobby project, resources are not available for large rewards.
Stickers or other small rewards are available. We reserve the right to refuse rewards for any reason.
Public recognition can be given upon request.
# Contact
Email: beardog [ at ] mailbox.org
PGP (optional): F61A 4DBB 0B3D F172 1F65 0EDF 0D41 4D0F E405 B63B

19
docs/README.md Normal file
View File

@ -0,0 +1,19 @@
# Onionr Documentation
The Onionr [whitepaper](whitepaper.md) is the best place to start both for users and developers.
## User Documentation
* [Installation](usage/install.md)
* [First steps](usage/firststeps.md)
* [Using Onionr Mail](usage/mail.md)
* [Using Onionr web pages](usage/pages.md)
* [Staying safe/anonymous](usage/safety.md)
## Developer Documentation
* [Development environment setup](dev/setup.md)
* [Technical overview](dev/overview.md)
* [Project layout](dev/layout.md)
* [Plugin development guide](dev/plugins.md)
* [Testing](dev/testing.md)

2
docs/api.md
View File

@ -1,2 +0,0 @@
HTTP API
TODO

118
docs/dev/http-api.md Executable file
View File

@ -0,0 +1,118 @@
# Onionr HTTP API
All HTTP interfaces in the Onionr reference client use the [Flask](http://flask.pocoo.org/) web framework with the [gevent](http://www.gevent.org/) WSGI server.
## Client & Public difference
The client API server is a locked down interface intended for authenticated local communication.
The public API server is available only remotely from Tor & I2P. It is the interface in which peers use to communicate with one another.
# Client API
Please note: endpoints that simply provide static web app files are not documented here.
* /serviceactive/pubkey
- Methods: GET
- Returns true or false based on if a given public key has an active direct connection service.
* /queueResponseAdd/key (DEPRECATED)
- Methods: POST
- Accepts form key 'data' to set queue response information from a plugin
- Returns success if no error occurs
* /queueResponse/key (DEPRECATED)
- Methods: GET
- Returns the queue response for a key. Returns failure with a 404 code if a code is not set.
* /ping
- Methods: GET
- Returns "pong!"
* /getblocksbytype/type
- Methods: GET
- Returns a list of stored blocks by a given type
* /getblockbody/hash
- Methods: GET
- Returns the main data section of a block
* /getblockdata/hash
- Methods: GET
- Returns the entire data contents of a block, including metadata.
* /getblockheader/hash
- Methods: GET
- Returns the header (metadata section) of a block.
* /hitcount
- Methods: GET
- Return the amount of requests the public api server has received this session
* /lastconnect
- Methods: GET
- Returns the epoch timestamp of when the last incoming connection to the public API server was logged
* /site/hash
- Methods: GET
- Returns HTML content out of a block
* /waitforshare/hash
- Methods: POST
- Prevents the public API server from listing or sharing a block until it has been uploaded to at least 1 peer.
* /shutdown
- Methods: GET
- Shutdown Onionr. You should probably use /shutdownclean instead.
* /shutdownclean
- Methods: GET
- Tells the communicator daemon to shutdown Onionr. Slower but cleaner.
* /getstats
- Methods: GET
- Returns some JSON serialized statistics
* /getuptime
- Methods: GET
- Returns uptime in seconds
* /getActivePubkey
- Methods: GET
- Returns the current active public key in base32 format
* /getHumanReadable/pubkey
- Methods: GET
- Echos the specified public key in mnemonic format
* /insertblock
- Methods: POST
- Accepts JSON data for creating a new block. 'message' contains the block data, 'to' specifies the peer's public key to encrypt the data to, 'sign' is a boolean for signing the message.
# Public API
v0
* /
- Methods: GET
- Returns a basic HTML informational banner describing Onionr.
* /getblocklist
- Methods: GET
- URI Parameters:
- date: unix epoch timestamp for offset
- Returns a list of block hashes stored on the node since an offset (all blocks if no timestamp is specified)
* /getdata/block-hash
- Methods: GET
- Returns data for a block based on a provided hash
* /www/file-path
- Methods: GET
- Returns file data. Intended for manually sharing file data directly from an Onionr node.
* /ping
- Methods: GET
- Returns 'pong!'
* /pex
- Methods: GET
- Returns a list of peer addresses reached within recent time
* /announce
- Methods: POST
- Accepts form data for 'node' (valid node address) and 'random' which is a nonce when hashed (blake2b_256) in the format `hash(peerAddress+serverAddress+nonce)`, begins with at least 5 zeros.
- Returns 200 with 'Success' if no error occurs. If the post is invalid, 'failure' with code 406 is returned.
* /upload
- Methods: POST
- Accepts form data for 'block' as a 'file' upload.
- Returns 200 with 'success' if no error occurs. If the block cannot be accepted, 'failure' with 400 is returned.
# Direct Connection API
These are constant endpoints available on direct connection servers. Plugin endpoints for direct connections are not documented here.
* /ping
- Methods: GET
- Returns 200 with 'pong!'
* /close
- Methods: GET
- Kills the direct connection server, destroying the onion address.
- Returns 200 with 'goodbye'

63
docs/dev/specs/block-spec.md Executable file
View File

@ -0,0 +1,63 @@
# Onionr Block Spec v1.1.0
# Block Description
Onionr Blocks are the primary means of sharing information in Onionr. Blocks are identified by a single hash value of their entire contents, using SHA3_256.
Blocks contain a JSON metadata section followed by a line break, with the main data section comprising the rest.
In the future, the specification will likely be updated to use flags and MessagePack instead of JSON with english keys.
# Encryption and Signatures
Onionr blocks may be encrypted or signed. In the reference client, this is done with libsodium, for both asymmetric and symmetric encryption.
Unlike many similar projects, blocks may completely be in plaintext, making Onionr suitable for sharing information publicly.
# Metadata Section
The metadata section has the following fields. If a block contains any other field, it must be considered invalid. All metadata fields are technically optional, but many are useful and essentially necessary for most use cases.
## meta
Max byte size: 1000
Meta is a string field which can contain arbitrary sub fields. It is intended for applications and plugins to use it for arbitrary metadata information. In the reference client, if the data section is encrypted or signed, the meta section also is.
Common meta fields, such as 'type' are used by the reference Onionr client to describe the type of a block.
## sig
Max byte size: 200
Sig is a field for storing public key signatures of the block, typically ed25519. In the reference client, this field is a base64 encoded signature of the meta field combined with the block data. (**Therefore, information outside of the meta and data fields cannot be trusted to be placed there by the signer, although it can still be assured that the particular block has not been modified.**)
Note: the max field size is larger than a EdDSA signature (which is what is typically used) in order to allow other primitives for signing in alternative implementations or future versions.
## signer
Max byte size: 200
Signer is a field for specifying the public key which signed the block. In the reference client this is a base64 encoded ed25519 public key.
## time
Max byte size: 10
Time is an integer field for specifying the time of which a block was created. The trustworthiness of this field is based on one's trust of the block creator, however blocks with a time field set in the future (past a reasonable clock skew) are thrown out by the reference client.
## expire
Max byte size: 10
Expire is an integer field for specifying the time of which the block creator has indicated that the block should be deleted. The purpose of this is for voluntarily freeing the burden of unwanted blocks on the Onionr network, rather than security/privacy (since blocks could be trivially kept past expiration). Regardless, the reference client deletes blocks after a preset time if the expire field is either not set or longer than the preset time.
## pow
Max byte size: 1000
Pow is a field for placing the nonce found to make a block meet a target proof of work. In theory, a block could meet a target without a random token in this field.
## encryptType
encryptType is a field to specify the mode of encryption for a block. The values supported by Onionr are 'asym' and 'sym'.

BIN
docs/network-comparison.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 63 KiB

BIN
docs/onionr-1.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 57 KiB

BIN
docs/onionr-2.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 30 KiB

After

Width:  |  Height:  |  Size: 34 KiB

BIN
docs/onionr-3.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 40 KiB

After

Width:  |  Height:  |  Size: 29 KiB

BIN
docs/onionr-icon.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 85 KiB

After

Width:  |  Height:  |  Size: 41 KiB

BIN
docs/onionr-logo.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 189 KiB

After

Width:  |  Height:  |  Size: 34 KiB

0
docs/tor-stinks-02.png Normal file → Executable file
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 28 KiB

After

Width:  |  Height:  |  Size: 28 KiB

7
docs/usage/firststeps.md Normal file
View File

@ -0,0 +1,7 @@
# Onionr First Steps
After installing Onionr, there are several things to do:
1. Setup a [deterministic address](usage/deterministic.md) (optional)
2. Add friends' ids
3. Publish your id

13
docs/usage/install.md Normal file
View File

@ -0,0 +1,13 @@
# Onionr Installation
The following steps work broadly speaking for Windows, Mac, and Linux.
1. Verify python3.6+ is installed: if its not see https://www.python.org/downloads/
2. Verify Tor is installed (does not need to be running, binary can be put into system path or Onionr directory)
3. [Optional but recommended]: setup virtual environment using [virtualenv](https://virtualenv.pypa.io/en/latest/), activate the virtual environment
4. Clone Onionr: git clone https://gitlab.com/beardog/onionr
5. Install the Python module dependencies: pip3 install --require-hashes -r requirements.txt

20
docs/whitepaper.md
View File

@ -39,11 +39,11 @@ When designing Onionr we had these main goals in mind:
At its core, Onionr is merely a description for storing data in self-verifying packages ("blocks"). These blocks can be encrypted to a user (or for one's self), encrypted symmetrically, or not at all. Blocks can be signed by their creator, but regardless, they are self-verifying due to being identified by a sha3-256 hash value; once a block is created, it cannot be modified.
Onionr exchanges a list of blocks between all nodes. By default, all nodes download and share all other blocks, however this is configurable. Blocks do not rely on any particular order of receipt or transport mechanism.
Onionr exchanges a list of blocks between all nodes. By default, all nodes download and share all other blocks, however, this is configurable. Blocks do not rely on any particular order of receipt or transport mechanism.
## User IDs
User IDs are simply Ed25519 public keys. They are represented in Base32 format, or encoded using the [PGP Word List](https://en.wikipedia.org/wiki/PGP_word_list).
User IDs are simply Ed25519 public keys. They are represented in Base32 format or encoded using the [PGP Word List](https://en.wikipedia.org/wiki/PGP_word_list).
Public keys can be generated deterministically with a password using a key derivation function (Argon2id). This password can be shared between many users in order to share data anonymously among a group, using only 1 password. This is useful in some cases, but is risky, as if one user causes the key to be compromised and does not notify the group or revoke the key, there is no way to know.
@ -53,7 +53,7 @@ Although Onionr is transport agnostic, the only supported transports in the refe
### Node Profiling
To mitigate maliciously slow or unreliable nodes, Onionr builds a profile on nodes it connects to. Nodes are assigned a score, which raises based on the amount of successful block transfers, speed, and reliability of a node, and reduces the score based on how unreliable a node is. If a node is unreachable for over 24 hours after contact, it is forgotten. Onionr can also prioritize connection to 'friend' nodes.
To mitigate maliciously slow or unreliable nodes, Onionr builds a profile on nodes it connects to. Nodes are assigned a score, which raises based on the number of successful block transfers, speed, and reliability of a node, and reduces the score based on how unreliable a node is. If a node is unreachable for over 24 hours after contact, it is forgotten. Onionr can also prioritize connections to 'friend' nodes.
## Block Format
@ -65,7 +65,7 @@ Optionally, a random token can be inserted into the metadata for use in Proof of
### Block Encryption
For encryption, Onionr uses ephemeral Curve25519 keys for key exchange and XSalsa20-Poly1305 as a symmetric cipher, or optionally using only XSalsa20-Poly1305 with a pre-shared key.
For encryption, Onionr uses ephemeral Curve25519 keys for key exchange and XSalsa20-Poly1305 as a symmetric cipher or optionally using only XSalsa20-Poly1305 with a pre-shared key.
Regardless of encryption, blocks can be signed internally using Ed25519.
@ -91,7 +91,7 @@ In addition, randomness beacons such as the one operated by [NIST](https://beaco
# Direct Connections
We propose a method of using Onionr's block sync system to enable direct connections between peers by having one peer request to connect to another using the peer's public key. Since the request is within a standard block, proof of work must be used to request connection. If the requested peer is available and wishes to accept the connection, Onionr will generate a temporary .onion address for the other peer to connect to. Alternatively, a reverse connection may be formed, which is faster to establish but requires a message brokering system instead of a standard socket.
We propose a method of using Onionr's block sync system to enable direct connections between peers by having one peer request to connect to another using the peer's public key. Since the request is within a standard block, proof of work must be used to request a connection. If the requested peer is available and wishes to accept the connection, Onionr will generate a temporary .onion address for the other peer to connect to. Alternatively, a reverse connection may be formed, which is faster to establish but requires a message brokering system instead of a standard socket.
The benefits of such a system are increased privacy, and the ability to anonymously communicate from multiple devices at once. In a traditional onion service, one's online status can be monitored and more easily correlated.
@ -119,7 +119,9 @@ We seek to protect the following information:
* Physical location/IP address of nodes on the network
* All block data from tampering
### Data we cannot or do not protect
### Unprotected Data
Onionr does not protect the following:
* Data specifically inserted as plaintext is available to the public
* The public key of signed plaintext blocks
@ -133,12 +135,6 @@ We assume that Tor onion services (v3) and I2P services cannot be trivially dean
Once quantum safe algorithms are more mature and have decent high level libraries, they will be deployed.
# Comparisons to other P2P software
Since Onionr is far from the first to implement many of these ideas (on their own), this section compares Onionr to other networks, using points we consider to be the most important.
![network comparison image](network-comparison.png)
# Conclusion
If successful, Onionr will be a complete decentralized platform for anonymous computing, complete with limited metadata exposure, both node and user anonymity, and spam prevention

0
install/onionr.service Normal file → Executable file
View File

602
onionr/api.py
View File

@ -1,602 +0,0 @@
'''
Onionr - P2P Anonymous Storage Network
This file handles all incoming http requests to the client, using Flask
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
from gevent.pywsgi import WSGIServer, WSGIHandler
from gevent import Timeout
import flask, cgi, uuid
from flask import request, Response, abort, send_from_directory
import sys, random, threading, hmac, base64, time, os, json, socket
import core
from onionrblockapi import Block
import onionrutils, onionrexceptions, onionrcrypto, blockimporter, onionrevents as events, logger, config
import httpapi
from httpapi import friendsapi, simplecache
import onionr
class FDSafeHandler(WSGIHandler):
'''Our WSGI handler. Doesn't do much non-default except timeouts'''
def handle(self):
timeout = Timeout(60, exception=Exception)
timeout.start()
try:
WSGIHandler.handle(self)
except Timeout as ex:
raise
def setBindIP(filePath):
'''Set a random localhost IP to a specified file (intended for private or public API localhost IPs)'''
if config.get('general.random_bind_ip', True):
hostOctets = [str(127), str(random.randint(0x02, 0xFF)), str(random.randint(0x02, 0xFF)), str(random.randint(0x02, 0xFF))]
data = '.'.join(hostOctets)
# Try to bind IP. Some platforms like Mac block non normal 127.x.x.x
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
s.bind((data, 0))
except OSError:
# if mac/non-bindable, show warning and default to 127.0.0.1
logger.warn('Your platform appears to not support random local host addresses 127.x.x.x. Falling back to 127.0.0.1.')
data = '127.0.0.1'
s.close()
else:
data = '127.0.0.1'
with open(filePath, 'w') as bindFile:
bindFile.write(data)
return data
class PublicAPI:
'''
The new client api server, isolated from the public api
'''
def __init__(self, clientAPI):
assert isinstance(clientAPI, API)
app = flask.Flask('PublicAPI')
self.i2pEnabled = config.get('i2p.host', False)
self.hideBlocks = [] # Blocks to be denied sharing
self.host = setBindIP(clientAPI._core.publicApiHostFile)
self.torAdder = clientAPI._core.hsAddress
self.i2pAdder = clientAPI._core.i2pAddress
self.bindPort = config.get('client.public.port')
self.lastRequest = 0
logger.info('Running public api on %s:%s' % (self.host, self.bindPort))
@app.before_request
def validateRequest():
'''Validate request has the correct hostname'''
# If high security level, deny requests to public (HS should be disabled anyway for Tor, but might not be for I2P)
if config.get('general.security_level', default=0) > 0:
abort(403)
if type(self.torAdder) is None and type(self.i2pAdder) is None:
# abort if our hs addresses are not known
abort(403)
if request.host not in (self.i2pAdder, self.torAdder):
# Disallow connection if wrong HTTP hostname, in order to prevent DNS rebinding attacks
abort(403)
@app.after_request
def sendHeaders(resp):
'''Send api, access control headers'''
resp.headers['Date'] = 'Thu, 1 Jan 1970 00:00:00 GMT' # Clock info is probably useful to attackers. Set to unix epoch, since we can't fully remove the header.
# CSP to prevent XSS. Mainly for client side attacks (if hostname protection could somehow be bypassed)
resp.headers["Content-Security-Policy"] = "default-src 'none'; script-src 'none'; object-src 'none'; style-src data: 'unsafe-inline'; img-src data:; media-src 'none'; frame-src 'none'; font-src 'none'; connect-src 'none'"
# Prevent click jacking
resp.headers['X-Frame-Options'] = 'deny'
# No sniff is possibly not needed
resp.headers['X-Content-Type-Options'] = "nosniff"
# Network API version
resp.headers['X-API'] = onionr.API_VERSION
# Close connections to limit FD use
resp.headers['Connection'] = "close"
self.lastRequest = clientAPI._core._utils.getRoundedEpoch(roundS=5)
return resp
@app.route('/')
def banner():
# Display a bit of information to people who visit a node address in their browser
try:
with open('static-data/index.html', 'r') as html:
resp = Response(html.read(), mimetype='text/html')
except FileNotFoundError:
resp = Response("")
return resp
@app.route('/getblocklist')
def getBlockList():
# Provide a list of our blocks, with a date offset
dateAdjust = request.args.get('date')
bList = clientAPI._core.getBlockList(dateRec=dateAdjust)
for b in self.hideBlocks:
if b in bList:
# Don't share blocks we created if they haven't been *uploaded* yet, makes it harder to find who created a block
bList.remove(b)
return Response('\n'.join(bList))
@app.route('/getdata/<name>')
def getBlockData(name):
# Share data for a block if we have it
resp = ''
data = name
if clientAPI._utils.validateHash(data):
if data not in self.hideBlocks:
if data in clientAPI._core.getBlockList():
block = clientAPI.getBlockData(data, raw=True)
try:
block = block.encode()
except AttributeError:
abort(404)
block = clientAPI._core._utils.strToBytes(block)
resp = block
#resp = base64.b64encode(block).decode()
if len(resp) == 0:
abort(404)
resp = ""
return Response(resp, mimetype='application/octet-stream')
@app.route('/www/<path:path>')
def wwwPublic(path):
# A way to share files directly over your .onion
if not config.get("www.public.run", True):
abort(403)
return send_from_directory(config.get('www.public.path', 'static-data/www/public/'), path)
@app.route('/ping')
def ping():
# Endpoint to test if nodes are up
return Response("pong!")
@app.route('/pex')
def peerExchange():
response = ','.join(clientAPI._core.listAdders(recent=3600))
if len(response) == 0:
response = ''
return Response(response)
@app.route('/announce', methods=['post'])
def acceptAnnounce():
resp = 'failure'
powHash = ''
randomData = ''
newNode = ''
ourAdder = clientAPI._core.hsAddress.encode()
try:
newNode = request.form['node'].encode()
except KeyError:
logger.warn('No node specified for upload')
pass
else:
try:
randomData = request.form['random']
randomData = base64.b64decode(randomData)
except KeyError:
logger.warn('No random data specified for upload')
else:
nodes = newNode + clientAPI._core.hsAddress.encode()
nodes = clientAPI._core._crypto.blake2bHash(nodes)
powHash = clientAPI._core._crypto.blake2bHash(randomData + nodes)
try:
powHash = powHash.decode()
except AttributeError:
pass
if powHash.startswith('0000'):
newNode = clientAPI._core._utils.bytesToStr(newNode)
if clientAPI._core._utils.validateID(newNode) and not newNode in clientAPI._core.onionrInst.communicatorInst.newPeers:
clientAPI._core.onionrInst.communicatorInst.newPeers.append(newNode)
resp = 'Success'
else:
logger.warn(newNode.decode() + ' failed to meet POW: ' + powHash)
resp = Response(resp)
return resp
@app.route('/upload', methods=['post'])
def upload():
'''Accept file uploads. In the future this will be done more often than on creation
to speed up block sync
'''
resp = 'failure'
try:
data = request.form['block']
except KeyError:
logger.warn('No block specified for upload')
pass
else:
if sys.getsizeof(data) < 100000000:
try:
if blockimporter.importBlockFromData(data, clientAPI._core):
resp = 'success'
else:
logger.warn('Error encountered importing uploaded block')
except onionrexceptions.BlacklistedBlock:
logger.debug('uploaded block is blacklisted')
pass
if resp == 'failure':
abort(400)
resp = Response(resp)
return resp
# Set instances, then startup our public api server
clientAPI.setPublicAPIInstance(self)
while self.torAdder == '':
clientAPI._core.refreshFirstStartVars()
self.torAdder = clientAPI._core.hsAddress
time.sleep(0.1)
self.httpServer = WSGIServer((self.host, self.bindPort), app, log=None, handler_class=FDSafeHandler)
self.httpServer.serve_forever()
class API:
'''
Client HTTP api
'''
callbacks = {'public' : {}, 'private' : {}}
def __init__(self, onionrInst, debug, API_VERSION):
'''
Initialize the api server, preping variables for later use
This initilization defines all of the API entry points and handlers for the endpoints and errors
This also saves the used host (random localhost IP address) to the data folder in host.txt
'''
self.debug = debug
self._core = onionrInst.onionrCore
self.startTime = self._core._utils.getEpoch()
self._crypto = onionrcrypto.OnionrCrypto(self._core)
self._utils = onionrutils.OnionrUtils(self._core)
app = flask.Flask(__name__)
bindPort = int(config.get('client.client.port', 59496))
self.bindPort = bindPort
# Be extremely mindful of this. These are endpoints available without a password
self.whitelistEndpoints = ('site', 'www', 'onionrhome', 'board', 'boardContent', 'sharedContent', 'mail', 'mailindex', 'friends', 'friendsindex')
self.clientToken = config.get('client.webpassword')
self.timeBypassToken = base64.b16encode(os.urandom(32)).decode()
self.publicAPI = None # gets set when the thread calls our setter... bad hack but kinda necessary with flask
#threading.Thread(target=PublicAPI, args=(self,)).start()
self.host = setBindIP(self._core.privateApiHostFile)
logger.info('Running api on %s:%s' % (self.host, self.bindPort))
self.httpServer = ''
self.pluginResponses = {} # Responses for plugin endpoints
self.queueResponse = {}
onionrInst.setClientAPIInst(self)
app.register_blueprint(friendsapi.friends)
app.register_blueprint(simplecache.simplecache)
httpapi.load_plugin_blueprints(app)
@app.before_request
def validateRequest():
'''Validate request has set password and is the correct hostname'''
# For the purpose of preventing DNS rebinding attacks
if request.host != '%s:%s' % (self.host, self.bindPort):
abort(403)
if request.endpoint in self.whitelistEndpoints:
return
try:
if not hmac.compare_digest(request.headers['token'], self.clientToken):
if not hmac.compare_digest(request.form['token'], self.clientToken):
abort(403)
except KeyError:
if not hmac.compare_digest(request.form['token'], self.clientToken):
abort(403)
@app.after_request
def afterReq(resp):
# Security headers
if request.endpoint == 'site':
resp.headers['Content-Security-Policy'] = "default-src 'none'; style-src data: 'unsafe-inline'; img-src data:"
else:
resp.headers['Content-Security-Policy'] = "default-src 'none'; script-src 'self'; object-src 'none'; style-src 'self'; img-src 'self'; media-src 'none'; frame-src 'none'; font-src 'none'; connect-src 'self'"
resp.headers['X-Frame-Options'] = 'deny'
resp.headers['X-Content-Type-Options'] = "nosniff"
resp.headers['Server'] = ''
resp.headers['Date'] = 'Thu, 1 Jan 1970 00:00:00 GMT' # Clock info is probably useful to attackers. Set to unix epoch.
resp.headers['Connection'] = "close"
return resp
@app.route('/board/', endpoint='board')
def loadBoard():
return send_from_directory('static-data/www/board/', "index.html")
@app.route('/mail/<path:path>', endpoint='mail')
def loadMail(path):
return send_from_directory('static-data/www/mail/', path)
@app.route('/mail/', endpoint='mailindex')
def loadMailIndex():
return send_from_directory('static-data/www/mail/', 'index.html')
@app.route('/friends/<path:path>', endpoint='friends')
def loadContacts(path):
return send_from_directory('static-data/www/friends/', path)
@app.route('/friends/', endpoint='friendsindex')
def loadContacts():
return send_from_directory('static-data/www/friends/', 'index.html')
@app.route('/board/<path:path>', endpoint='boardContent')
def boardContent(path):
return send_from_directory('static-data/www/board/', path)
@app.route('/shared/<path:path>', endpoint='sharedContent')
def sharedContent(path):
return send_from_directory('static-data/www/shared/', path)
@app.route('/www/<path:path>', endpoint='www')
def wwwPublic(path):
if not config.get("www.private.run", True):
abort(403)
return send_from_directory(config.get('www.private.path', 'static-data/www/private/'), path)
@app.route('/queueResponseAdd/<name>', methods=['post'])
def queueResponseAdd(name):
# Responses from the daemon. TODO: change to direct var access instead of http endpoint
self.queueResponse[name] = request.form['data']
return Response('success')
@app.route('/queueResponse/<name>')
def queueResponse(name):
# Fetch a daemon queue response
resp = 'failure'
try:
resp = self.queueResponse[name]
except KeyError:
pass
else:
del self.queueResponse[name]
return Response(resp)
@app.route('/ping')
def ping():
# Used to check if client api is working
return Response("pong!")
@app.route('/', endpoint='onionrhome')
def hello():
# ui home
return send_from_directory('static-data/www/private/', 'index.html')
@app.route('/getblocksbytype/<name>')
def getBlocksByType(name):
blocks = self._core.getBlocksByType(name)
return Response(','.join(blocks))
@app.route('/getblockbody/<name>')
def getBlockBodyData(name):
resp = ''
if self._core._utils.validateHash(name):
try:
resp = Block(name, decrypt=True).bcontent
#resp = cgi.escape(Block(name, decrypt=True).bcontent, quote=True)
except TypeError:
pass
else:
abort(404)
return Response(resp)
@app.route('/getblockdata/<name>')
def getData(name):
resp = ""
if self._core._utils.validateHash(name):
if name in self._core.getBlockList():
try:
resp = self.getBlockData(name, decrypt=True)
except ValueError:
pass
else:
abort(404)
else:
abort(404)
return Response(resp)
@app.route('/getblockheader/<name>')
def getBlockHeader(name):
resp = self.getBlockData(name, decrypt=True, headerOnly=True)
return Response(resp)
@app.route('/lastconnect')
def lastConnect():
return Response(str(self.publicAPI.lastRequest))
@app.route('/site/<name>', endpoint='site')
def site(name):
bHash = name
resp = 'Not Found'
if self._core._utils.validateHash(bHash):
try:
resp = Block(bHash).bcontent
except onionrexceptions.NoDataAvailable:
abort(404)
except TypeError:
pass
try:
resp = base64.b64decode(resp)
except:
pass
if resp == 'Not Found' or not resp:
abort(404)
return Response(resp)
@app.route('/waitforshare/<name>', methods=['post'])
def waitforshare(name):
'''Used to prevent the **public** api from sharing blocks we just created'''
assert name.isalnum()
if name in self.publicAPI.hideBlocks:
self.publicAPI.hideBlocks.remove(name)
return Response("removed")
else:
self.publicAPI.hideBlocks.append(name)
return Response("added")
@app.route('/shutdown')
def shutdown():
try:
self.publicAPI.httpServer.stop()
self.httpServer.stop()
except AttributeError:
pass
return Response("bye")
@app.route('/shutdownclean')
def shutdownClean():
# good for calling from other clients
self._core.daemonQueueAdd('shutdown')
return Response("bye")
@app.route('/getstats')
def getStats():
# returns node stats
#return Response("disabled")
while True:
try:
return Response(self._core.serializer.getStats())
except AttributeError:
pass
@app.route('/getuptime')
def showUptime():
return Response(str(self.getUptime()))
@app.route('/getActivePubkey')
def getActivePubkey():
return Response(self._core._crypto.pubKey)
@app.route('/getHumanReadable/<name>')
def getHumanReadable(name):
return Response(self._core._utils.getHumanReadableID(name))
@app.route('/insertblock', methods=['POST'])
def insertBlock():
encrypt = False
bData = request.get_json(force=True)
message = bData['message']
subject = 'temp'
encryptType = ''
sign = True
meta = {}
to = ''
try:
if bData['encrypt']:
to = bData['to']
encrypt = True
encryptType = 'asym'
except KeyError:
pass
try:
if not bData['sign']:
sign = False
except KeyError:
pass
try:
bType = bData['type']
except KeyError:
bType = 'bin'
try:
meta = json.loads(bData['meta'])
except KeyError:
pass
threading.Thread(target=self._core.insertBlock, args=(message,), kwargs={'header': bType, 'encryptType': encryptType, 'sign':sign, 'asymPeer': to, 'meta': meta}).start()
return Response('success')
@app.route('/apipoints/<path:subpath>', methods=['POST', 'GET'])
def pluginEndpoints(subpath=''):
'''Send data to plugins'''
# TODO have a variable for the plugin to set data to that we can use for the response
pluginResponseCode = str(uuid.uuid4())
resp = 'success'
responseTimeout = 20
startTime = self._core._utils.getEpoch()
postData = {}
if request.method == 'POST':
postData = request.form['postData']
if len(subpath) > 1:
data = subpath.split('/')
if len(data) > 1:
plName = data[0]
events.event('pluginRequest', {'name': plName, 'path': subpath, 'pluginResponse': pluginResponseCode, 'postData': postData}, onionr=onionrInst)
while True:
try:
resp = self.pluginResponses[pluginResponseCode]
except KeyError:
time.sleep(0.2)
if self._core._utils.getEpoch() - startTime > responseTimeout:
abort(504)
break
else:
break
else:
abort(404)
return Response(resp)
self.httpServer = WSGIServer((self.host, bindPort), app, log=None, handler_class=FDSafeHandler)
self.httpServer.serve_forever()
def setPublicAPIInstance(self, inst):
assert isinstance(inst, PublicAPI)
self.publicAPI = inst
def validateToken(self, token):
'''
Validate that the client token matches the given token. Used to prevent CSRF and data exfiltration
'''
if len(self.clientToken) == 0:
logger.error("client password needs to be set")
return False
try:
if not hmac.compare_digest(self.clientToken, token):
return False
else:
return True
except TypeError:
return False
def getUptime(self):
while True:
try:
return self._utils.getEpoch - startTime
except AttributeError:
# Don't error on race condition with startup
pass
def getBlockData(self, bHash, decrypt=False, raw=False, headerOnly=False):
assert self._core._utils.validateHash(bHash)
bl = Block(bHash, core=self._core)
if decrypt:
bl.decrypt()
if bl.isEncrypted and not bl.decrypted:
raise ValueError
if not raw:
if not headerOnly:
retData = {'meta':bl.bheader, 'metadata': bl.bmetadata, 'content': bl.bcontent}
for x in list(retData.keys()):
try:
retData[x] = retData[x].decode()
except AttributeError:
pass
else:
validSig = False
signer = self._core._utils.bytesToStr(bl.signer)
#print(signer, bl.isSigned(), self._core._utils.validatePubKey(signer), bl.isSigner(signer))
if bl.isSigned() and self._core._utils.validatePubKey(signer) and bl.isSigner(signer):
validSig = True
bl.bheader['validSig'] = validSig
bl.bheader['meta'] = ''
retData = {'meta': bl.bheader, 'metadata': bl.bmetadata}
return json.dumps(retData)
else:
return bl.raw

3
onionr/apiservers/__init__.py Executable file
View File

@ -0,0 +1,3 @@
from . import public, private
PublicAPI = public.PublicAPI
ClientAPI = private.PrivateAPI

96
onionr/apiservers/private/__init__.py Normal file
View File

@ -0,0 +1,96 @@
'''
Onionr - Private P2P Communication
This file handles all incoming http requests to the client, using Flask
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import base64, os
import flask
from gevent.pywsgi import WSGIServer
import logger
from onionrutils import epoch
import httpapi
from . import register_private_blueprints
class PrivateAPI:
'''
Client HTTP api
'''
callbacks = {'public' : {}, 'private' : {}}
def __init__(self, onionrInst, debug, API_VERSION):
'''
Initialize the api server, preping variables for later use
This initialization defines all of the API entry points and handlers for the endpoints and errors
This also saves the used host (random localhost IP address) to the data folder in host.txt
'''
config = onionrInst.config
self.config = config
self.debug = debug
self._core = onionrInst.onionrCore
self.startTime = epoch.get_epoch()
self._crypto = self._core._crypto
app = flask.Flask(__name__)
bindPort = int(config.get('client.client.port', 59496))
self.bindPort = bindPort
self.clientToken = config.get('client.webpassword')
self.timeBypassToken = base64.b16encode(os.urandom(32)).decode()
self.publicAPI = None # gets set when the thread calls our setter... bad hack but kinda necessary with flask
#threading.Thread(target=PublicAPI, args=(self,)).start()
self.host = httpapi.apiutils.setbindip.set_bind_IP(self._core.privateApiHostFile, self._core)
logger.info('Running api on %s:%s' % (self.host, self.bindPort))
self.httpServer = ''
self.queueResponse = {}
self.get_block_data = httpapi.apiutils.GetBlockData(self)
onionrInst.setClientAPIInst(self)
register_private_blueprints.register_private_blueprints(self, app)
httpapi.load_plugin_blueprints(app)
self.httpServer = WSGIServer((self.host, bindPort), app, log=None, handler_class=httpapi.fdsafehandler.FDSafeHandler)
self.httpServer.serve_forever()
def setPublicAPIInstance(self, inst):
self.publicAPI = inst
def validateToken(self, token):
'''
Validate that the client token matches the given token. Used to prevent CSRF and data exfiltration
'''
if len(self.clientToken) == 0:
logger.error("client password needs to be set")
return False
try:
if not hmac.compare_digest(self.clientToken, token):
return False
else:
return True
except TypeError:
return False
def getUptime(self):
while True:
try:
return epoch.get_epoch() - self.startTime
except (AttributeError, NameError):
# Don't error on race condition with startup
pass
def getBlockData(self, bHash, decrypt=False, raw=False, headerOnly=False):
return self.get_block_data.get_block_data(bHash, decrypt=decrypt, raw=raw, headerOnly=headerOnly)

33
onionr/apiservers/private/register_private_blueprints.py Normal file
View File

@ -0,0 +1,33 @@
'''
Onionr - Private P2P Communication
This file registers blueprints for the private api server
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import os
from httpapi import security, friendsapi, profilesapi, configapi, insertblock, miscclientapi, onionrsitesapi, apiutils
def register_private_blueprints(private_api, app):
app.register_blueprint(security.client.ClientAPISecurity(private_api).client_api_security_bp)
app.register_blueprint(friendsapi.friends)
app.register_blueprint(profilesapi.profile_BP)
app.register_blueprint(configapi.config_BP)
app.register_blueprint(insertblock.ib)
app.register_blueprint(miscclientapi.getblocks.client_get_blocks)
app.register_blueprint(miscclientapi.endpoints.PrivateEndpoints(private_api).private_endpoints_bp)
app.register_blueprint(onionrsitesapi.site_api)
app.register_blueprint(apiutils.shutdown.shutdown_bp)
app.register_blueprint(miscclientapi.staticfiles.static_files_bp)
return app

56
onionr/apiservers/public/__init__.py Normal file
View File

@ -0,0 +1,56 @@
'''
Onionr - Private P2P Communication
This file handles all incoming http requests to the public api server, using Flask
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import time
import flask
from gevent.pywsgi import WSGIServer
from httpapi import apiutils, security, fdsafehandler, miscpublicapi
import logger, onionr
class PublicAPI:
'''
The new client api server, isolated from the public api
'''
def __init__(self, clientAPI):
config = clientAPI.config
app = flask.Flask('PublicAPI')
app.config['MAX_CONTENT_LENGTH'] = 5 * 1024 * 1024
self.i2pEnabled = config.get('i2p.host', False)
self.hideBlocks = [] # Blocks to be denied sharing
self.host = apiutils.setbindip.set_bind_IP(clientAPI._core.publicApiHostFile, clientAPI._core)
self.torAdder = clientAPI._core.hsAddress
self.i2pAdder = clientAPI._core.i2pAddress
self.bindPort = config.get('client.public.port')
self.lastRequest = 0
self.hitCount = 0 # total rec requests to public api since server started
self.config = config
self.clientAPI = clientAPI
self.API_VERSION = onionr.API_VERSION
logger.info('Running public api on %s:%s' % (self.host, self.bindPort))
# Set instances, then startup our public api server
clientAPI.setPublicAPIInstance(self)
while self.torAdder == '':
clientAPI._core.refreshFirstStartVars()
self.torAdder = clientAPI._core.hsAddress
time.sleep(0.1)
app.register_blueprint(security.public.PublicAPISecurity(self).public_api_security_bp)
app.register_blueprint(miscpublicapi.endpoints.PublicEndpoints(self).public_endpoints_bp)
self.httpServer = WSGIServer((self.host, self.bindPort), app, log=None, handler_class=fdsafehandler.FDSafeHandler)
self.httpServer.serve_forever()

11
onionr/blockimporter.py
View File

@ -1,5 +1,5 @@
'''
Onionr - P2P Microblogging Platform & Social network
Onionr - Private P2P Communication
Import block data and save it
'''
@ -18,6 +18,7 @@
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import core, onionrexceptions, logger
from onionrutils import validatemetadata, blockmetadata
def importBlockFromData(content, coreInst):
retData = False
@ -34,17 +35,17 @@ def importBlockFromData(content, coreInst):
except AttributeError:
pass
metas = coreInst._utils.getBlockMetadataFromData(content) # returns tuple(metadata, meta), meta is also in metadata
metas = blockmetadata.get_block_metadata_from_data(content) # returns tuple(metadata, meta), meta is also in metadata
metadata = metas[0]
if coreInst._utils.validateMetadata(metadata, metas[2]): # check if metadata is valid
if validatemetadata.validate_metadata(coreInst, metadata, metas[2]): # check if metadata is valid
if coreInst._crypto.verifyPow(content): # check if POW is enough/correct
logger.info('Block passed proof, saving.')
logger.info('Block passed proof, saving.', terminal=True)
try:
blockHash = coreInst.setData(content)
except onionrexceptions.DiskAllocationReached:
pass
else:
coreInst.addToBlockDB(blockHash, dataSaved=True)
coreInst._utils.processBlockMetadata(blockHash) # caches block metadata values to block database
blockmetadata.process_block_metadata(coreInst, blockHash) # caches block metadata values to block database
retData = True
return retData

646
onionr/communicator.py
View File

@ -1,646 +0,0 @@
#!/usr/bin/env python3
'''
Onionr - P2P Anonymous Storage Network
This file contains both the OnionrCommunicate class for communcating with peers
and code to operate as a daemon, getting commands from the command queue database (see core.Core.daemonQueue)
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sys, os, core, config, json, requests, time, logger, threading, base64, onionr, uuid, binascii
from dependencies import secrets
from utils import networkmerger
import onionrexceptions, onionrpeers, onionrevents as events, onionrplugins as plugins, onionrblockapi as block
from communicatorutils import onionrdaemontools
import onionrsockets, onionr, onionrproofs
from communicatorutils import onionrcommunicatortimers, proxypicker
OnionrCommunicatorTimers = onionrcommunicatortimers.OnionrCommunicatorTimers
config.reload()
class OnionrCommunicatorDaemon:
def __init__(self, onionrInst, proxyPort, developmentMode=config.get('general.dev_mode', False)):
onionrInst.communicatorInst = self
# configure logger and stuff
onionr.Onionr.setupConfig('data/', self = self)
self.proxyPort = proxyPort
self.isOnline = True # Assume we're connected to the internet
# list of timer instances
self.timers = []
# initialize core with Tor socks port being 3rd argument
self.proxyPort = proxyPort
self._core = onionrInst.onionrCore
self.blocksToUpload = []
# loop time.sleep delay in seconds
self.delay = 1
# lists of connected peers and peers we know we can't reach currently
self.onlinePeers = []
self.offlinePeers = []
self.cooldownPeer = {}
self.connectTimes = {}
self.peerProfiles = [] # list of peer's profiles (onionrpeers.PeerProfile instances)
self.newPeers = [] # Peers merged to us. Don't add to db until we know they're reachable
# amount of threads running by name, used to prevent too many
self.threadCounts = {}
# set true when shutdown command received
self.shutdown = False
# list of new blocks to download, added to when new block lists are fetched from peers
self.blockQueue = {}
# list of blocks currently downloading, avoid s
self.currentDownloading = []
# timestamp when the last online node was seen
self.lastNodeSeen = None
# Dict of time stamps for peer's block list lookup times, to avoid downloading full lists all the time
self.dbTimestamps = {}
# Clear the daemon queue for any dead messages
if os.path.exists(self._core.queueDB):
self._core.clearDaemonQueue()
# Loads in and starts the enabled plugins
plugins.reload()
# daemon tools are misc daemon functions, e.g. announce to online peers
# intended only for use by OnionrCommunicatorDaemon
self.daemonTools = onionrdaemontools.DaemonTools(self)
# time app started running for info/statistics purposes
self.startTime = self._core._utils.getEpoch()
if developmentMode:
OnionrCommunicatorTimers(self, self.heartbeat, 30)
# Set timers, function reference, seconds
# requiresPeer True means the timer function won't fire if we have no connected peers
peerPoolTimer = OnionrCommunicatorTimers(self, self.getOnlinePeers, 60, maxThreads=1)
OnionrCommunicatorTimers(self, self.runCheck, 2, maxThreads=1)
OnionrCommunicatorTimers(self, self.lookupBlocks, self._core.config.get('timers.lookupBlocks'), requiresPeer=True, maxThreads=1)
OnionrCommunicatorTimers(self, self.getBlocks, self._core.config.get('timers.getBlocks'), requiresPeer=True, maxThreads=2)
OnionrCommunicatorTimers(self, self.clearOfflinePeer, 58)
blockCleanupTimer = OnionrCommunicatorTimers(self, self.daemonTools.cleanOldBlocks, 65)
OnionrCommunicatorTimers(self, self.lookupAdders, 60, requiresPeer=True)
OnionrCommunicatorTimers(self, self.daemonTools.cooldownPeer, 30, requiresPeer=True)
OnionrCommunicatorTimers(self, self.uploadBlock, 10, requiresPeer=True, maxThreads=1)
OnionrCommunicatorTimers(self, self.daemonCommands, 6, maxThreads=1)
OnionrCommunicatorTimers(self, self.detectAPICrash, 30, maxThreads=1)
deniableBlockTimer = OnionrCommunicatorTimers(self, self.daemonTools.insertDeniableBlock, 180, requiresPeer=True, maxThreads=1)
netCheckTimer = OnionrCommunicatorTimers(self, self.daemonTools.netCheck, 600)
if config.get('general.security_level') == 0:
announceTimer = OnionrCommunicatorTimers(self, self.daemonTools.announceNode, 3600, requiresPeer=True, maxThreads=1)
announceTimer.count = (announceTimer.frequency - 120)
else:
logger.debug('Will not announce node.')
cleanupTimer = OnionrCommunicatorTimers(self, self.peerCleanup, 300, requiresPeer=True)
forwardSecrecyTimer = OnionrCommunicatorTimers(self, self.daemonTools.cleanKeys, 15, maxThreads=1)
# set loop to execute instantly to load up peer pool (replaced old pool init wait)
peerPoolTimer.count = (peerPoolTimer.frequency - 1)
cleanupTimer.count = (cleanupTimer.frequency - 60)
deniableBlockTimer.count = (deniableBlockTimer.frequency - 175)
blockCleanupTimer.count = (blockCleanupTimer.frequency - 5)
#forwardSecrecyTimer.count = (forwardSecrecyTimer.frequency - 990)
if config.get('general.socket_servers'):
self.socketServer = threading.Thread(target=onionrsockets.OnionrSocketServer, args=(self._core,))
self.socketServer.start()
self.socketClient = onionrsockets.OnionrSocketClient(self._core)
# Main daemon loop, mainly for calling timers, don't do any complex operations here to avoid locking
try:
while not self.shutdown:
for i in self.timers:
if self.shutdown:
break
i.processTimer()
time.sleep(self.delay)
# Debug to print out used FDs (regular and net)
#proc = psutil.Process()
#print(proc.open_files(), len(psutil.net_connections()))
except KeyboardInterrupt:
self.shutdown = True
pass
logger.info('Goodbye.')
self._core.killSockets = True
self._core._utils.localCommand('shutdown') # shutdown the api
time.sleep(0.5)
def lookupAdders(self):
'''Lookup new peer addresses'''
logger.info('Looking up new addresses...')
tryAmount = 1
newPeers = []
for i in range(tryAmount):
# Download new peer address list from random online peers
if len(newPeers) > 10000:
# Dont get new peers if we have too many queued up
break
peer = self.pickOnlinePeer()
newAdders = self.peerAction(peer, action='pex')
try:
newPeers = newAdders.split(',')
except AttributeError:
pass
else:
# Validate new peers are good format and not already in queue
invalid = []
for x in newPeers:
x = x.strip()
if not self._core._utils.validateID(x) or x in self.newPeers or x == self._core.hsAddress:
invalid.append(x)
for x in invalid:
newPeers.remove(x)
self.newPeers.extend(newPeers)
self.decrementThreadCount('lookupAdders')
def lookupBlocks(self):
'''Lookup new blocks & add them to download queue'''
logger.info('Looking up new blocks...')
tryAmount = 2
newBlocks = ''
existingBlocks = self._core.getBlockList()
triedPeers = [] # list of peers we've tried this time around
maxBacklog = 1560 # Max amount of *new* block hashes to have already in queue, to avoid memory exhaustion
lastLookupTime = 0 # Last time we looked up a particular peer's list
for i in range(tryAmount):
listLookupCommand = 'getblocklist' # This is defined here to reset it each time
if len(self.blockQueue) >= maxBacklog:
break
if not self.isOnline:
break
# check if disk allocation is used
if self._core._utils.storageCounter.isFull():
logger.debug('Not looking up new blocks due to maximum amount of allowed disk space used')
break
peer = self.pickOnlinePeer() # select random online peer
# if we've already tried all the online peers this time around, stop
if peer in triedPeers:
if len(self.onlinePeers) == len(triedPeers):
break
else:
continue
triedPeers.append(peer)
# Get the last time we looked up a peer's stamp to only fetch blocks since then.
# Saved in memory only for privacy reasons
try:
lastLookupTime = self.dbTimestamps[peer]
except KeyError:
lastLookupTime = 0
else:
listLookupCommand += '?date=%s' % (lastLookupTime,)
try:
newBlocks = self.peerAction(peer, listLookupCommand) # get list of new block hashes
except Exception as error:
logger.warn('Could not get new blocks from %s.' % peer, error = error)
newBlocks = False
else:
self.dbTimestamps[peer] = self._core._utils.getRoundedEpoch(roundS=60)
if newBlocks != False:
# if request was a success
for i in newBlocks.split('\n'):
if self._core._utils.validateHash(i):
# if newline seperated string is valid hash
if not i in existingBlocks:
# if block does not exist on disk and is not already in block queue
if i not in self.blockQueue:
if onionrproofs.hashMeetsDifficulty(i) and not self._core._blacklist.inBlacklist(i):
if len(self.blockQueue) <= 1000000:
self.blockQueue[i] = [peer] # add blocks to download queue
else:
if peer not in self.blockQueue[i]:
if len(self.blockQueue[i]) < 10:
self.blockQueue[i].append(peer)
self.decrementThreadCount('lookupBlocks')
return
def getBlocks(self):
'''download new blocks in queue'''
for blockHash in list(self.blockQueue):
triedQueuePeers = [] # List of peers we've tried for a block
try:
blockPeers = list(self.blockQueue[blockHash])
except KeyError:
blockPeers = []
removeFromQueue = True
if self.shutdown or not self.isOnline:
# Exit loop if shutting down or offline
break
# Do not download blocks being downloaded or that are already saved (edge cases)
if blockHash in self.currentDownloading:
#logger.debug('Already downloading block %s...' % blockHash)
continue
if blockHash in self._core.getBlockList():
#logger.debug('Block %s is already saved.' % (blockHash,))
try:
del self.blockQueue[blockHash]
except KeyError:
pass
continue
if self._core._blacklist.inBlacklist(blockHash):
continue
if self._core._utils.storageCounter.isFull():
break
self.currentDownloading.append(blockHash) # So we can avoid concurrent downloading in other threads of same block
if len(blockPeers) == 0:
peerUsed = self.pickOnlinePeer()
else:
blockPeers = self._core._crypto.randomShuffle(blockPeers)
peerUsed = blockPeers.pop(0)
if not self.shutdown and peerUsed.strip() != '':
logger.info("Attempting to download %s from %s..." % (blockHash[:12], peerUsed))
content = self.peerAction(peerUsed, 'getdata/' + blockHash) # block content from random peer (includes metadata)
if content != False and len(content) > 0:
try:
content = content.encode()
except AttributeError:
pass
realHash = self._core._crypto.sha3Hash(content)
try:
realHash = realHash.decode() # bytes on some versions for some reason
except AttributeError:
pass
if realHash == blockHash:
content = content.decode() # decode here because sha3Hash needs bytes above
metas = self._core._utils.getBlockMetadataFromData(content) # returns tuple(metadata, meta), meta is also in metadata
metadata = metas[0]
if self._core._utils.validateMetadata(metadata, metas[2]): # check if metadata is valid, and verify nonce
if self._core._crypto.verifyPow(content): # check if POW is enough/correct
logger.info('Attempting to save block %s...' % blockHash[:12])
try:
self._core.setData(content)
except onionrexceptions.DiskAllocationReached:
logger.error('Reached disk allocation allowance, cannot save block %s.' % blockHash)
removeFromQueue = False
else:
self._core.addToBlockDB(blockHash, dataSaved=True)
self._core._utils.processBlockMetadata(blockHash) # caches block metadata values to block database
else:
logger.warn('POW failed for block %s.' % blockHash)
else:
if self._core._blacklist.inBlacklist(realHash):
logger.warn('Block %s is blacklisted.' % (realHash,))
else:
logger.warn('Metadata for block %s is invalid.' % blockHash)
self._core._blacklist.addToDB(blockHash)
else:
# if block didn't meet expected hash
tempHash = self._core._crypto.sha3Hash(content) # lazy hack, TODO use var
try:
tempHash = tempHash.decode()
except AttributeError:
pass
# Punish peer for sharing invalid block (not always malicious, but is bad regardless)
onionrpeers.PeerProfiles(peerUsed, self._core).addScore(-50)
if tempHash != 'ed55e34cb828232d6c14da0479709bfa10a0923dca2b380496e6b2ed4f7a0253':
# Dumb hack for 404 response from peer. Don't log it if 404 since its likely not malicious or a critical error.
logger.warn('Block hash validation failed for ' + blockHash + ' got ' + tempHash)
else:
removeFromQueue = False # Don't remove from queue if 404
if removeFromQueue:
try:
del self.blockQueue[blockHash] # remove from block queue both if success or false
except KeyError:
pass
self.currentDownloading.remove(blockHash)
self.decrementThreadCount('getBlocks')
return
def pickOnlinePeer(self):
'''randomly picks peer from pool without bias (using secrets module)'''
retData = ''
while True:
peerLength = len(self.onlinePeers)
if peerLength <= 0:
break
try:
# get a random online peer, securely. May get stuck in loop if network is lost or if all peers in pool magically disconnect at once
retData = self.onlinePeers[self._core._crypto.secrets.randbelow(peerLength)]
except IndexError:
pass
else:
break
return retData
def decrementThreadCount(self, threadName):
'''Decrement amount of a thread name if more than zero, called when a function meant to be run in a thread ends'''
try:
if self.threadCounts[threadName] > 0:
self.threadCounts[threadName] -= 1
except KeyError:
pass
def clearOfflinePeer(self):
'''Removes the longest offline peer to retry later'''
try:
removed = self.offlinePeers.pop(0)
except IndexError:
pass
else:
logger.debug('Removed ' + removed + ' from offline list, will try them again.')
self.decrementThreadCount('clearOfflinePeer')
def getOnlinePeers(self):
'''
Manages the self.onlinePeers attribute list, connects to more peers if we have none connected
'''
logger.debug('Refreshing peer pool...')
maxPeers = int(config.get('peers.max_connect', 10))
needed = maxPeers - len(self.onlinePeers)
for i in range(needed):
if len(self.onlinePeers) == 0:
self.connectNewPeer(useBootstrap=True)
else:
self.connectNewPeer()
if self.shutdown:
break
else:
if len(self.onlinePeers) == 0:
logger.debug('Couldn\'t connect to any peers.' + (' Last node seen %s ago.' % self.daemonTools.humanReadableTime(time.time() - self.lastNodeSeen) if not self.lastNodeSeen is None else ''))
else:
self.lastNodeSeen = time.time()
self.decrementThreadCount('getOnlinePeers')
def addBootstrapListToPeerList(self, peerList):
'''
Add the bootstrap list to the peer list (no duplicates)
'''
for i in self._core.bootstrapList:
if i not in peerList and i not in self.offlinePeers and i != self._core.hsAddress and len(str(i).strip()) > 0:
peerList.append(i)
self._core.addAddress(i)
def connectNewPeer(self, peer='', useBootstrap=False):
'''Adds a new random online peer to self.onlinePeers'''
retData = False
tried = self.offlinePeers
if peer != '':
if self._core._utils.validateID(peer):
peerList = [peer]
else:
raise onionrexceptions.InvalidAddress('Will not attempt connection test to invalid address')
else:
peerList = self._core.listAdders()
mainPeerList = self._core.listAdders()
peerList = onionrpeers.getScoreSortedPeerList(self._core)
if len(peerList) < 8 or secrets.randbelow(4) == 3:
tryingNew = []
for x in self.newPeers:
if x not in peerList:
peerList.append(x)
tryingNew.append(x)
for i in tryingNew:
self.newPeers.remove(i)
if len(peerList) == 0 or useBootstrap:
# Avoid duplicating bootstrap addresses in peerList
self.addBootstrapListToPeerList(peerList)
for address in peerList:
if not config.get('tor.v3onions') and len(address) == 62:
continue
if address == self._core.hsAddress:
continue
if len(address) == 0 or address in tried or address in self.onlinePeers or address in self.cooldownPeer:
continue
if self.shutdown:
return
if self.peerAction(address, 'ping') == 'pong!':
time.sleep(0.1)
if address not in mainPeerList:
networkmerger.mergeAdders(address, self._core)
if address not in self.onlinePeers:
logger.info('Connected to ' + address)
self.onlinePeers.append(address)
self.connectTimes[address] = self._core._utils.getEpoch()
retData = address
# add peer to profile list if they're not in it
for profile in self.peerProfiles:
if profile.address == address:
break
else:
self.peerProfiles.append(onionrpeers.PeerProfiles(address, self._core))
break
else:
tried.append(address)
logger.debug('Failed to connect to ' + address)
return retData
def removeOnlinePeer(self, peer):
'''Remove an online peer'''
try:
del self.connectTimes[peer]
except KeyError:
pass
try:
del self.dbTimestamps[peer]
except KeyError:
pass
try:
self.onlinePeers.remove(peer)
except ValueError:
pass
def peerCleanup(self):
'''This just calls onionrpeers.cleanupPeers, which removes dead or bad peers (offline too long, too slow)'''
onionrpeers.peerCleanup(self._core)
self.decrementThreadCount('peerCleanup')
def printOnlinePeers(self):
'''logs online peer list'''
if len(self.onlinePeers) == 0:
logger.warn('No online peers')
else:
logger.info('Online peers:')
for i in self.onlinePeers:
score = str(self.getPeerProfileInstance(i).score)
logger.info(i + ', score: ' + score)
def peerAction(self, peer, action, data='', returnHeaders=False):
'''Perform a get request to a peer'''
if len(peer) == 0:
return False
#logger.debug('Performing ' + action + ' with ' + peer + ' on port ' + str(self.proxyPort))
url = 'http://%s/%s' % (peer, action)
if len(data) > 0:
url += '&data=' + data
self._core.setAddressInfo(peer, 'lastConnectAttempt', self._core._utils.getEpoch()) # mark the time we're trying to request this peer
retData = self._core._utils.doGetRequest(url, port=self.proxyPort)
# if request failed, (error), mark peer offline
if retData == False:
try:
self.getPeerProfileInstance(peer).addScore(-10)
self.removeOnlinePeer(peer)
if action != 'ping':
self.getOnlinePeers() # Will only add a new peer to pool if needed
except ValueError:
pass
else:
self._core.setAddressInfo(peer, 'lastConnect', self._core._utils.getEpoch())
self.getPeerProfileInstance(peer).addScore(1)
return retData # If returnHeaders, returns tuple of data, headers. if not, just data string
def getPeerProfileInstance(self, peer):
'''Gets a peer profile instance from the list of profiles, by address name'''
for i in self.peerProfiles:
# if the peer's profile is already loaded, return that
if i.address == peer:
retData = i
break
else:
# if the peer's profile is not loaded, return a new one. connectNewPeer adds it the list on connect
retData = onionrpeers.PeerProfiles(peer, self._core)
return retData
def getUptime(self):
return self._core._utils.getEpoch() - self.startTime
def heartbeat(self):
'''Show a heartbeat debug message'''
logger.debug('Heartbeat. Node running for %s.' % self.daemonTools.humanReadableTime(self.getUptime()))
self.decrementThreadCount('heartbeat')
def daemonCommands(self):
'''
Process daemon commands from daemonQueue
'''
cmd = self._core.daemonQueue()
response = ''
if cmd is not False:
events.event('daemon_command', onionr = None, data = {'cmd' : cmd})
if cmd[0] == 'shutdown':
self.shutdown = True
elif cmd[0] == 'announceNode':
if len(self.onlinePeers) > 0:
self.announce(cmd[1])
else:
logger.debug("No nodes connected. Will not introduce node.")
elif cmd[0] == 'runCheck': # deprecated
logger.debug('Status check; looks good.')
open(self._core.dataDir + '.runcheck', 'w+').close()
elif cmd[0] == 'connectedPeers':
response = '\n'.join(list(self.onlinePeers)).strip()
if response == '':
response = 'none'
elif cmd[0] == 'localCommand':
response = self._core._utils.localCommand(cmd[1])
elif cmd[0] == 'pex':
for i in self.timers:
if i.timerFunction.__name__ == 'lookupAdders':
i.count = (i.frequency - 1)
elif cmd[0] == 'uploadBlock':
self.blocksToUpload.append(cmd[1])
elif cmd[0] == 'startSocket':
# Create our own socket server
socketInfo = json.loads(cmd[1])
socketInfo['id'] = uuid.uuid4()
self._core.startSocket = socketInfo
elif cmd[0] == 'addSocket':
# Socket server was created for us
socketInfo = json.loads(cmd[1])
peer = socketInfo['peer']
reason = socketInfo['reason']
threading.Thread(target=self.socketClient.startSocket, args=(peer, reason)).start()
else:
logger.info('Recieved daemonQueue command:' + cmd[0])
if cmd[0] not in ('', None):
if response != '':
self._core._utils.localCommand('queueResponseAdd/' + cmd[4], post=True, postData={'data': response})
response = ''
self.decrementThreadCount('daemonCommands')
def uploadBlock(self):
'''Upload our block to a few peers'''
# when inserting a block, we try to upload it to a few peers to add some deniability
triedPeers = []
finishedUploads = []
self.blocksToUpload = self._core._crypto.randomShuffle(self.blocksToUpload)
if len(self.blocksToUpload) != 0:
for bl in self.blocksToUpload:
if not self._core._utils.validateHash(bl):
logger.warn('Requested to upload invalid block')
self.decrementThreadCount('uploadBlock')
return
for i in range(min(len(self.onlinePeers), 6)):
peer = self.pickOnlinePeer()
if peer in triedPeers:
continue
triedPeers.append(peer)
url = 'http://' + peer + '/upload'
data = {'block': block.Block(bl).getRaw()}
proxyType = proxypicker.pick_proxy(peer)
logger.info("Uploading block to " + peer)
if not self._core._utils.doPostRequest(url, data=data, proxyType=proxyType) == False:
self._core._utils.localCommand('waitforshare/' + bl, post=True)
finishedUploads.append(bl)
for x in finishedUploads:
try:
self.blocksToUpload.remove(x)
except ValueError:
pass
self.decrementThreadCount('uploadBlock')
def announce(self, peer):
'''Announce to peers our address'''
if self.daemonTools.announceNode() == False:
logger.warn('Could not introduce node.')
def detectAPICrash(self):
'''exit if the api server crashes/stops'''
if self._core._utils.localCommand('ping', silent=False) not in ('pong', 'pong!'):
for i in range(8):
if self._core._utils.localCommand('ping') in ('pong', 'pong!') or self.shutdown:
break # break for loop
time.sleep(1)
else:
# This executes if the api is NOT detected to be running
events.event('daemon_crash', onionr = None, data = {})
logger.error('Daemon detected API crash (or otherwise unable to reach API after long time), stopping...')
self.shutdown = True
self.decrementThreadCount('detectAPICrash')
def runCheck(self):
if self.daemonTools.runCheck():
logger.debug('Status check; looks good.')
self.decrementThreadCount('runCheck')
def startCommunicator(onionrInst, proxyPort):
OnionrCommunicatorDaemon(onionrInst, proxyPort)

254
onionr/communicator/__init__.py Executable file
View File

@ -0,0 +1,254 @@
'''
Onionr - Private P2P Communication
This file contains both the OnionrCommunicate class for communcating with peers
and code to operate as a daemon, getting commands from the command queue database (see core.Core.daemonQueue)
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sys, os, time
import core, config, logger, onionr
import onionrexceptions, onionrpeers, onionrevents as events, onionrplugins as plugins, onionrblockapi as block
from . import onlinepeers
from communicatorutils import servicecreator, onionrcommunicatortimers
from communicatorutils import downloadblocks, lookupblocks, lookupadders
from communicatorutils import servicecreator, connectnewpeers, uploadblocks
from communicatorutils import daemonqueuehandler, announcenode, deniableinserts
from communicatorutils import cooldownpeer, housekeeping, netcheck
from onionrutils import localcommand, epoch
from etc import humanreadabletime
import onionrservices, onionr, onionrproofs
OnionrCommunicatorTimers = onionrcommunicatortimers.OnionrCommunicatorTimers
config.reload()
class OnionrCommunicatorDaemon:
def __init__(self, onionrInst, proxyPort, developmentMode=config.get('general.dev_mode', False)):
onionrInst.communicatorInst = self
# configure logger and stuff
onionr.Onionr.setupConfig('data/', self = self)
self.proxyPort = proxyPort
self.isOnline = True # Assume we're connected to the internet
# list of timer instances
self.timers = []
# initialize core with Tor socks port being 3rd argument
self.proxyPort = proxyPort
self._core = onionrInst.onionrCore
self.blocksToUpload = []
# loop time.sleep delay in seconds
self.delay = 1
# lists of connected peers and peers we know we can't reach currently
self.onlinePeers = []
self.offlinePeers = []
self.cooldownPeer = {}
self.connectTimes = {}
self.peerProfiles = [] # list of peer's profiles (onionrpeers.PeerProfile instances)
self.newPeers = [] # Peers merged to us. Don't add to db until we know they're reachable
self.announceProgress = {}
self.announceCache = {}
# amount of threads running by name, used to prevent too many
self.threadCounts = {}
# set true when shutdown command received
self.shutdown = False
# list of new blocks to download, added to when new block lists are fetched from peers
self.blockQueue = {}
# list of blocks currently downloading, avoid s
self.currentDownloading = []
# timestamp when the last online node was seen
self.lastNodeSeen = None
# Dict of time stamps for peer's block list lookup times, to avoid downloading full lists all the time
self.dbTimestamps = {}
# Clear the daemon queue for any dead messages
if os.path.exists(self._core.queueDB):
self._core.clearDaemonQueue()
# Loads in and starts the enabled plugins
plugins.reload()
# time app started running for info/statistics purposes
self.startTime = epoch.get_epoch()
if developmentMode:
OnionrCommunicatorTimers(self, self.heartbeat, 30)
# Set timers, function reference, seconds
# requiresPeer True means the timer function won't fire if we have no connected peers
peerPoolTimer = OnionrCommunicatorTimers(self, onlinepeers.get_online_peers, 60, maxThreads=1, myArgs=[self])
OnionrCommunicatorTimers(self, self.runCheck, 2, maxThreads=1)
# Timers to periodically lookup new blocks and download them
OnionrCommunicatorTimers(self, self.lookupBlocks, self._core.config.get('timers.lookupBlocks', 25), requiresPeer=True, maxThreads=1)
OnionrCommunicatorTimers(self, self.getBlocks, self._core.config.get('timers.getBlocks', 30), requiresPeer=True, maxThreads=2)
# Timer to reset the longest offline peer so contact can be attempted again
OnionrCommunicatorTimers(self, onlinepeers.clear_offline_peer, 58, myArgs=[self])
# Timer to cleanup old blocks
blockCleanupTimer = OnionrCommunicatorTimers(self, housekeeping.clean_old_blocks, 65, myArgs=[self])
# Timer to discover new peers
OnionrCommunicatorTimers(self, self.lookupAdders, 60, requiresPeer=True)
# Timer for adjusting which peers we actively communicate to at any given time, to avoid over-using peers
OnionrCommunicatorTimers(self, cooldownpeer.cooldown_peer, 30, myArgs=[self], requiresPeer=True)
# Timer to read the upload queue and upload the entries to peers
OnionrCommunicatorTimers(self, uploadblocks.upload_blocks_from_communicator, 5, myArgs=[self], requiresPeer=True, maxThreads=1)
# Timer to process the daemon command queue
OnionrCommunicatorTimers(self, daemonqueuehandler.handle_daemon_commands, 6, myArgs=[self], maxThreads=3)
# Setup direct connections
if config.get('general.socket_servers', False):
self.services = onionrservices.OnionrServices(self._core)
self.active_services = []
self.service_greenlets = []
OnionrCommunicatorTimers(self, servicecreator.service_creator, 5, maxThreads=50, myArgs=[self])
else:
self.services = None
# This timer creates deniable blocks, in an attempt to further obfuscate block insertion metadata
if config.get('general.insert_deniable_blocks', True):
deniableBlockTimer = OnionrCommunicatorTimers(self, deniableinserts.insert_deniable_block, 180, myArgs=[self], requiresPeer=True, maxThreads=1)
deniableBlockTimer.count = (deniableBlockTimer.frequency - 175)
# Timer to check for connectivity, through Tor to various high-profile onion services
netCheckTimer = OnionrCommunicatorTimers(self, netcheck.net_check, 600, myArgs=[self])
# Announce the public API server transport address to other nodes if security level allows
if config.get('general.security_level', 1) == 0 and config.get('general.announce_node', True):
# Default to high security level incase config breaks
announceTimer = OnionrCommunicatorTimers(self, announcenode.announce_node, 3600, myArgs=[self], requiresPeer=True, maxThreads=1)
announceTimer.count = (announceTimer.frequency - 120)
else:
logger.debug('Will not announce node.')
# Timer to delete malfunctioning or long-dead peers
cleanupTimer = OnionrCommunicatorTimers(self, self.peerCleanup, 300, requiresPeer=True)
# Timer to cleanup dead ephemeral forward secrecy keys
forwardSecrecyTimer = OnionrCommunicatorTimers(self, housekeeping.clean_keys, 15, myArgs=[self], maxThreads=1)
# Adjust initial timer triggers
peerPoolTimer.count = (peerPoolTimer.frequency - 1)
cleanupTimer.count = (cleanupTimer.frequency - 60)
blockCleanupTimer.count = (blockCleanupTimer.frequency - 5)
# Main daemon loop, mainly for calling timers, don't do any complex operations here to avoid locking
try:
while not self.shutdown:
for i in self.timers:
if self.shutdown:
break
i.processTimer()
time.sleep(self.delay)
# Debug to print out used FDs (regular and net)
#proc = psutil.Process()
#print(proc.open_files(), len(psutil.net_connections()))
except KeyboardInterrupt:
self.shutdown = True
pass
logger.info('Goodbye. (Onionr is cleaning up, and will exit)', terminal=True)
try:
self.service_greenlets
except AttributeError:
pass
else:
for server in self.service_greenlets:
server.stop()
localcommand.local_command(self._core, 'shutdown') # shutdown the api
time.sleep(0.5)
def lookupAdders(self):
'''Lookup new peer addresses'''
lookupadders.lookup_new_peer_transports_with_communicator(self)
def lookupBlocks(self):
'''Lookup new blocks & add them to download queue'''
lookupblocks.lookup_blocks_from_communicator(self)
def getBlocks(self):
'''download new blocks in queue'''
downloadblocks.download_blocks_from_communicator(self)
def decrementThreadCount(self, threadName):
'''Decrement amount of a thread name if more than zero, called when a function meant to be run in a thread ends'''
try:
if self.threadCounts[threadName] > 0:
self.threadCounts[threadName] -= 1
except KeyError:
pass
def connectNewPeer(self, peer='', useBootstrap=False):
'''Adds a new random online peer to self.onlinePeers'''
connectnewpeers.connect_new_peer_to_communicator(self, peer, useBootstrap)
def peerCleanup(self):
'''This just calls onionrpeers.cleanupPeers, which removes dead or bad peers (offline too long, too slow)'''
onionrpeers.peer_cleanup(self._core)
self.decrementThreadCount('peerCleanup')
def getPeerProfileInstance(self, peer):
'''Gets a peer profile instance from the list of profiles, by address name'''
for i in self.peerProfiles:
# if the peer's profile is already loaded, return that
if i.address == peer:
retData = i
break
else:
# if the peer's profile is not loaded, return a new one. connectNewPeer adds it the list on connect
retData = onionrpeers.PeerProfiles(peer, self._core)
return retData
def getUptime(self):
return epoch.get_epoch() - self.startTime
def heartbeat(self):
'''Show a heartbeat debug message'''
logger.debug('Heartbeat. Node running for %s.' % humanreadabletime.human_readable_time(self.getUptime()))
self.decrementThreadCount('heartbeat')
def announce(self, peer):
'''Announce to peers our address'''
if announcenode.announce_node(self) == False:
logger.warn('Could not introduce node.', terminal=True)
def runCheck(self):
if run_file_exists(self):
logger.debug('Status check; looks good.')
self.decrementThreadCount('runCheck')
def startCommunicator(onionrInst, proxyPort):
OnionrCommunicatorDaemon(onionrInst, proxyPort)
def run_file_exists(daemon):
if os.path.isfile(daemon._core.dataDir + '.runcheck'):
os.remove(daemon._core.dataDir + '.runcheck')
return True
return False

27
onionr/communicator/bootstrappeers.py Normal file
View File

@ -0,0 +1,27 @@
'''
Onionr - Private P2P Communication
add bootstrap peers to the communicator peer list
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
def add_bootstrap_list_to_peer_list(comm_inst, peerList):
'''
Add the bootstrap list to the peer list (no duplicates)
'''
for i in comm_inst._core.bootstrapList:
if i not in peerList and i not in comm_inst.offlinePeers and i != comm_inst._core.hsAddress and len(str(i).strip()) > 0:
peerList.append(i)
comm_inst._core.addAddress(i)

6
onionr/communicator/onlinepeers/__init__.py Normal file
View File

@ -0,0 +1,6 @@
from . import clearofflinepeer, onlinepeers, pickonlinepeers, removeonlinepeer
clear_offline_peer = clearofflinepeer.clear_offline_peer
get_online_peers = onlinepeers.get_online_peers
pick_online_peer = pickonlinepeers.pick_online_peer
remove_online_peer = removeonlinepeer.remove_online_peer

29
onionr/communicator/onlinepeers/clearofflinepeer.py Normal file
View File

@ -0,0 +1,29 @@
'''
Onionr - Private P2P Communication
clear offline peer in a communicator instance
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import logger
def clear_offline_peer(comm_inst):
'''Removes the longest offline peer to retry later'''
try:
removed = comm_inst.offlinePeers.pop(0)
except IndexError:
pass
else:
logger.debug('Removed ' + removed + ' from offline list, will try them again.')
comm_inst.decrementThreadCount('clear_offline_peer')

45
onionr/communicator/onlinepeers/onlinepeers.py Normal file
View File

@ -0,0 +1,45 @@
'''
Onionr - Private P2P Communication
get online peers in a communicator instance
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import time
from etc import humanreadabletime
import logger
def get_online_peers(comm_inst):
'''
Manages the comm_inst.onlinePeers attribute list, connects to more peers if we have none connected
'''
config = comm_inst._core.config
logger.debug('Refreshing peer pool...')
maxPeers = int(config.get('peers.max_connect', 10))
needed = maxPeers - len(comm_inst.onlinePeers)
for i in range(needed):
if len(comm_inst.onlinePeers) == 0:
comm_inst.connectNewPeer(useBootstrap=True)
else:
comm_inst.connectNewPeer()
if comm_inst.shutdown:
break
else:
if len(comm_inst.onlinePeers) == 0:
logger.debug('Couldn\'t connect to any peers.' + (' Last node seen %s ago.' % humanreadabletime.human_readable_time(time.time() - comm_inst.lastNodeSeen) if not comm_inst.lastNodeSeen is None else ''), terminal=True)
else:
comm_inst.lastNodeSeen = time.time()
comm_inst.decrementThreadCount('get_online_peers')

34
onionr/communicator/onlinepeers/pickonlinepeers.py Normal file
View File

@ -0,0 +1,34 @@
'''
Onionr - Private P2P Communication
pick online peers in a communicator instance
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
def pick_online_peer(comm_inst):
'''randomly picks peer from pool without bias (using secrets module)'''
retData = ''
while True:
peerLength = len(comm_inst.onlinePeers)
if peerLength <= 0:
break
try:
# get a random online peer, securely. May get stuck in loop if network is lost or if all peers in pool magically disconnect at once
retData = comm_inst.onlinePeers[comm_inst._core._crypto.secrets.randbelow(peerLength)]
except IndexError:
pass
else:
break
return retData

33
onionr/communicator/onlinepeers/removeonlinepeer.py Normal file
View File

@ -0,0 +1,33 @@
'''
Onionr - Private P2P Communication
remove an online peer from the pool in a communicator instance
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
def remove_online_peer(comm_inst, peer):
'''Remove an online peer'''
try:
del comm_inst.connectTimes[peer]
except KeyError:
pass
try:
del comm_inst.dbTimestamps[peer]
except KeyError:
pass
try:
comm_inst.onlinePeers.remove(peer)
except ValueError:
pass

53
onionr/communicator/peeraction.py Normal file
View File

@ -0,0 +1,53 @@
'''
Onionr - Private P2P Communication
This file implements logic for performing requests to Onionr peers
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import streamedrequests
import logger
from onionrutils import epoch, basicrequests
from . import onlinepeers
def peer_action(comm_inst, peer, action, data='', returnHeaders=False, max_resp_size=5242880):
'''Perform a get request to a peer'''
penalty_score = -10
if len(peer) == 0:
return False
url = 'http://%s/%s' % (peer, action)
if len(data) > 0:
url += '&data=' + data
comm_inst._core.setAddressInfo(peer, 'lastConnectAttempt', epoch.get_epoch()) # mark the time we're trying to request this peer
try:
retData = basicrequests.do_get_request(comm_inst._core, url, port=comm_inst.proxyPort, max_size=max_resp_size)
except streamedrequests.exceptions.ResponseLimitReached:
logger.warn('Request failed due to max response size being overflowed', terminal=True)
retData = False
penalty_score = -100
# if request failed, (error), mark peer offline
if retData == False:
try:
comm_inst.getPeerProfileInstance(peer).addScore(penalty_score)
onlinepeers.remove_online_peer(comm_inst, peer)
if action != 'ping' and not comm_inst.shutdown:
logger.warn('Lost connection to ' + peer, terminal=True)
onlinepeers.get_online_peers(comm_inst) # Will only add a new peer to pool if needed
except ValueError:
pass
else:
comm_inst._core.setAddressInfo(peer, 'lastConnect', epoch.get_epoch())
comm_inst.getPeerProfileInstance(peer).addScore(1)
return retData # If returnHeaders, returns tuple of data, headers. if not, just data string

33
onionr/communicatorutils/README.md Executable file
View File

@ -0,0 +1,33 @@
# communicatorutils
The files in this submodule handle various subtasks and utilities for the onionr communicator.
## Files:
announcenode.py: Uses a communicator instance to announce our transport address to connected nodes
connectnewpeers.py: takes a communicator instance and has it connect to as many peers as needed, and/or to a new specified peer.
cooldownpeer.py: randomly selects a connected peer in a communicator and disconnects them for the purpose of security and network balancing.
daemonqueuehandler.py: checks for new commands in the daemon queue and processes them accordingly.
deniableinserts.py: insert fake blocks with the communicator for plausible deniability
downloadblocks.py: iterates a communicator instance's block download queue and attempts to download the blocks from online peers
housekeeping.py: cleans old blocks and forward secrecy keys
lookupadders.py: ask connected peers to share their list of peer transport addresses
lookupblocks.py: lookup new blocks from connected peers from the communicator
netcheck.py: check if the node is online based on communicator status and onion server ping results
onionrcommunicataortimers.py: create a timer for a function to be launched on an interval. Control how many possible instances of a timer may be running a function at once and control if the timer should be ran in a thread or not.
proxypicker.py: returns a string name for the appropriate proxy to be used with a particular peer transport address.
servicecreator.py: iterate connection blocks and create new direct connection servers for them.
uploadblocks.py: iterate a communicator's upload queue and upload the blocks to connected peers

0
onionr/communicatorutils/__init__.py Executable file
View File

86
onionr/communicatorutils/announcenode.py Executable file
View File

@ -0,0 +1,86 @@
'''
Onionr - Private P2P Communication
Use a communicator instance to announce our transport address to connected nodes
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import base64
import onionrproofs, logger
from etc import onionrvalues
from onionrutils import basicrequests, bytesconverter
from communicator import onlinepeers
def announce_node(daemon):
'''Announce our node to our peers'''
ov = onionrvalues.OnionrValues()
retData = False
announceFail = False
# Do not let announceCache get too large
if len(daemon.announceCache) >= 10000:
daemon.announceCache.popitem()
if daemon._core.config.get('general.security_level', 0) == 0:
# Announce to random online peers
for i in daemon.onlinePeers:
if not i in daemon.announceCache and not i in daemon.announceProgress:
peer = i
break
else:
peer = onlinepeers.pick_online_peer(daemon)
for x in range(1):
if x == 1 and daemon._core.config.get('i2p.host'):
ourID = daemon._core.config.get('i2p.own_addr').strip()
else:
ourID = daemon._core.hsAddress.strip()
url = 'http://' + peer + '/announce'
data = {'node': ourID}
combinedNodes = ourID + peer
if ourID != 1:
#TODO: Extend existingRand for i2p
existingRand = bytesconverter.bytes_to_str(daemon._core.getAddressInfo(peer, 'powValue'))
# Reset existingRand if it no longer meets the minimum POW
if type(existingRand) is type(None) or not existingRand.endswith('0' * ov.announce_pow):
existingRand = ''
if peer in daemon.announceCache:
data['random'] = daemon.announceCache[peer]
elif len(existingRand) > 0:
data['random'] = existingRand
else:
daemon.announceProgress[peer] = True
proof = onionrproofs.DataPOW(combinedNodes, forceDifficulty=ov.announce_pow)
del daemon.announceProgress[peer]
try:
data['random'] = base64.b64encode(proof.waitForResult()[1])
except TypeError:
# Happens when we failed to produce a proof
logger.error("Failed to produce a pow for announcing to " + peer)
announceFail = True
else:
daemon.announceCache[peer] = data['random']
if not announceFail:
logger.info('Announcing node to ' + url)
if basicrequests.do_post_request(daemon._core, url, data) == 'Success':
logger.info('Successfully introduced node to ' + peer, terminal=True)
retData = True
daemon._core.setAddressInfo(peer, 'introduced', 1)
daemon._core.setAddressInfo(peer, 'powValue', data['random'])
daemon.decrementThreadCount('announce_node')
return retData

89
onionr/communicatorutils/connectnewpeers.py Executable file
View File

@ -0,0 +1,89 @@
'''
Onionr - Private P2P Communication
Connect a new peer to our communicator instance. Does so randomly if no peer is specified
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import time, sys, secrets
import onionrexceptions, logger, onionrpeers
from utils import networkmerger
from onionrutils import stringvalidators, epoch
from communicator import peeraction, bootstrappeers
def connect_new_peer_to_communicator(comm_inst, peer='', useBootstrap=False):
config = comm_inst._core.config
retData = False
tried = comm_inst.offlinePeers
if peer != '':
if stringvalidators.validate_transport(peer):
peerList = [peer]
else:
raise onionrexceptions.InvalidAddress('Will not attempt connection test to invalid address')
else:
peerList = comm_inst._core.listAdders()
mainPeerList = comm_inst._core.listAdders()
peerList = onionrpeers.get_score_sorted_peer_list(comm_inst._core)
# If we don't have enough peers connected or random chance, select new peers to try
if len(peerList) < 8 or secrets.randbelow(4) == 3:
tryingNew = []
for x in comm_inst.newPeers:
if x not in peerList:
peerList.append(x)
tryingNew.append(x)
for i in tryingNew:
comm_inst.newPeers.remove(i)
if len(peerList) == 0 or useBootstrap:
# Avoid duplicating bootstrap addresses in peerList
bootstrappeers.add_bootstrap_list_to_peer_list(comm_inst, peerList)
for address in peerList:
if not config.get('tor.v3onions') and len(address) == 62:
continue
# Don't connect to our own address
if address == comm_inst._core.hsAddress:
continue
# Don't connect to invalid address or if its already been tried/connected, or if its cooled down
if len(address) == 0 or address in tried or address in comm_inst.onlinePeers or address in comm_inst.cooldownPeer:
continue
if comm_inst.shutdown:
return
# Ping a peer,
if peeraction.peer_action(comm_inst, address, 'ping') == 'pong!':
time.sleep(0.1)
if address not in mainPeerList:
# Add a peer to our list if it isn't already since it successfully connected
networkmerger.mergeAdders(address, comm_inst._core)
if address not in comm_inst.onlinePeers:
logger.info('Connected to ' + address, terminal=True)
comm_inst.onlinePeers.append(address)
comm_inst.connectTimes[address] = epoch.get_epoch()
retData = address
# add peer to profile list if they're not in it
for profile in comm_inst.peerProfiles:
if profile.address == address:
break
else:
comm_inst.peerProfiles.append(onionrpeers.PeerProfiles(address, comm_inst._core))
break
else:
# Mark a peer as tried if they failed to respond to ping
tried.append(address)
logger.debug('Failed to connect to ' + address)
return retData

53
onionr/communicatorutils/cooldownpeer.py Executable file
View File

@ -0,0 +1,53 @@
'''
Onionr - Private P2P Communication
Select a random online peer in a communicator instance and have them "cool down"
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
from onionrutils import epoch
from communicator import onlinepeers
def cooldown_peer(comm_inst):
'''Randomly add an online peer to cooldown, so we can connect a new one'''
onlinePeerAmount = len(comm_inst.onlinePeers)
minTime = 300
cooldownTime = 600
toCool = ''
tempConnectTimes = dict(comm_inst.connectTimes)
# Remove peers from cooldown that have been there long enough
tempCooldown = dict(comm_inst.cooldownPeer)
for peer in tempCooldown:
if (epoch.get_epoch() - tempCooldown[peer]) >= cooldownTime:
del comm_inst.cooldownPeer[peer]
# Cool down a peer, if we have max connections alive for long enough
if onlinePeerAmount >= comm_inst._core.config.get('peers.max_connect', 10, save = True):
finding = True
while finding:
try:
toCool = min(tempConnectTimes, key=tempConnectTimes.get)
if (epoch.get_epoch() - tempConnectTimes[toCool]) < minTime:
del tempConnectTimes[toCool]
else:
finding = False
except ValueError:
break
else:
onlinepeers.remove_online_peer(comm_inst, toCool)
comm_inst.cooldownPeer[toCool] = epoch.get_epoch()
comm_inst.decrementThreadCount('cooldown_peer')

56
onionr/communicatorutils/daemonqueuehandler.py Executable file
View File

@ -0,0 +1,56 @@
'''
Onionr - P2P Anonymous Storage Network
Handle daemon queue commands in the communicator
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import logger
import onionrevents as events
from onionrutils import localcommand
def handle_daemon_commands(comm_inst):
cmd = comm_inst._core.daemonQueue()
response = ''
if cmd is not False:
events.event('daemon_command', onionr = comm_inst._core.onionrInst, data = {'cmd' : cmd})
if cmd[0] == 'shutdown':
comm_inst.shutdown = True
elif cmd[0] == 'announceNode':
if len(comm_inst.onlinePeers) > 0:
comm_inst.announce(cmd[1])
else:
logger.debug("No nodes connected. Will not introduce node.")
elif cmd[0] == 'runCheck': # deprecated
logger.debug('Status check; looks good.')
open(comm_inst._core.dataDir + '.runcheck', 'w+').close()
elif cmd[0] == 'connectedPeers':
response = '\n'.join(list(comm_inst.onlinePeers)).strip()
if response == '':
response = 'none'
elif cmd[0] == 'localCommand':
response = localcommand.local_command(comm_inst._core, cmd[1])
elif cmd[0] == 'pex':
for i in comm_inst.timers:
if i.timerFunction.__name__ == 'lookupAdders':
i.count = (i.frequency - 1)
elif cmd[0] == 'uploadBlock':
comm_inst.blocksToUpload.append(cmd[1])
if cmd[0] not in ('', None):
if response != '':
localcommand.local_command(comm_inst._core, 'queueResponseAdd/' + cmd[4], post=True, postData={'data': response})
response = ''
comm_inst.decrementThreadCount('handle_daemon_commands')

31
onionr/communicatorutils/deniableinserts.py Executable file
View File

@ -0,0 +1,31 @@
'''
Onionr - Private P2P Communication
Use the communicator to insert fake mail messages
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import secrets
from etc import onionrvalues
def insert_deniable_block(comm_inst):
'''Insert a fake block in order to make it more difficult to track real blocks'''
fakePeer = ''
chance = 10
if secrets.randbelow(chance) == (chance - 1):
# This assumes on the libsodium primitives to have key-privacy
fakePeer = onionrvalues.DENIABLE_PEER_ADDRESS
data = secrets.token_hex(secrets.randbelow(1024) + 1)
comm_inst._core.insertBlock(data, header='pm', encryptType='asym', asymPeer=fakePeer, disableForward=True, meta={'subject': 'foo'})
comm_inst.decrementThreadCount('insert_deniable_block')

116
onionr/communicatorutils/downloadblocks/__init__.py Executable file
View File

@ -0,0 +1,116 @@
'''
Onionr - Private P2P Communication
Download blocks using the communicator instance
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import communicator, onionrexceptions
import logger, onionrpeers
from onionrutils import blockmetadata, stringvalidators, validatemetadata
from . import shoulddownload
from communicator import peeraction, onlinepeers
def download_blocks_from_communicator(comm_inst):
assert isinstance(comm_inst, communicator.OnionrCommunicatorDaemon)
for blockHash in list(comm_inst.blockQueue):
if len(comm_inst.onlinePeers) == 0:
break
triedQueuePeers = [] # List of peers we've tried for a block
try:
blockPeers = list(comm_inst.blockQueue[blockHash])
except KeyError:
blockPeers = []
removeFromQueue = True
if not shoulddownload.should_download(comm_inst, blockHash):
continue
if comm_inst.shutdown or not comm_inst.isOnline or comm_inst._core.storage_counter.isFull():
# Exit loop if shutting down or offline, or disk allocation reached
break
# Do not download blocks being downloaded
if blockHash in comm_inst.currentDownloading:
#logger.debug('Already downloading block %s...' % blockHash)
continue
comm_inst.currentDownloading.append(blockHash) # So we can avoid concurrent downloading in other threads of same block
if len(blockPeers) == 0:
peerUsed = onlinepeers.pick_online_peer(comm_inst)
else:
blockPeers = comm_inst._core._crypto.randomShuffle(blockPeers)
peerUsed = blockPeers.pop(0)
if not comm_inst.shutdown and peerUsed.strip() != '':
logger.info("Attempting to download %s from %s..." % (blockHash[:12], peerUsed))
content = peeraction.peer_action(comm_inst, peerUsed, 'getdata/' + blockHash, max_resp_size=3000000) # block content from random peer (includes metadata)
if content != False and len(content) > 0:
try:
content = content.encode()
except AttributeError:
pass
realHash = comm_inst._core._crypto.sha3Hash(content)
try:
realHash = realHash.decode() # bytes on some versions for some reason
except AttributeError:
pass
if realHash == blockHash:
content = content.decode() # decode here because sha3Hash needs bytes above
metas = blockmetadata.get_block_metadata_from_data(content) # returns tuple(metadata, meta), meta is also in metadata
metadata = metas[0]
if validatemetadata.validate_metadata(comm_inst._core, metadata, metas[2]): # check if metadata is valid, and verify nonce
if comm_inst._core._crypto.verifyPow(content): # check if POW is enough/correct
logger.info('Attempting to save block %s...' % blockHash[:12])
try:
comm_inst._core.setData(content)
except onionrexceptions.DataExists:
logger.warn('Data is already set for %s ' % (blockHash,))
except onionrexceptions.DiskAllocationReached:
logger.error('Reached disk allocation allowance, cannot save block %s.' % (blockHash,))
removeFromQueue = False
else:
comm_inst._core.addToBlockDB(blockHash, dataSaved=True)
blockmetadata.process_block_metadata(comm_inst._core, blockHash) # caches block metadata values to block database
else:
logger.warn('POW failed for block %s.' % (blockHash,))
else:
if comm_inst._core._blacklist.inBlacklist(realHash):
logger.warn('Block %s is blacklisted.' % (realHash,))
else:
logger.warn('Metadata for block %s is invalid.' % (blockHash,))
comm_inst._core._blacklist.addToDB(blockHash)
else:
# if block didn't meet expected hash
tempHash = comm_inst._core._crypto.sha3Hash(content) # lazy hack, TODO use var
try:
tempHash = tempHash.decode()
except AttributeError:
pass
# Punish peer for sharing invalid block (not always malicious, but is bad regardless)
onionrpeers.PeerProfiles(peerUsed, comm_inst._core).addScore(-50)
if tempHash != 'ed55e34cb828232d6c14da0479709bfa10a0923dca2b380496e6b2ed4f7a0253':
# Dumb hack for 404 response from peer. Don't log it if 404 since its likely not malicious or a critical error.
logger.warn('Block hash validation failed for ' + blockHash + ' got ' + tempHash)
else:
removeFromQueue = False # Don't remove from queue if 404
if removeFromQueue:
try:
del comm_inst.blockQueue[blockHash] # remove from block queue both if success or false
logger.info('%s blocks remaining in queue' % [len(comm_inst.blockQueue)], terminal=True)
except KeyError:
pass
comm_inst.currentDownloading.remove(blockHash)
comm_inst.decrementThreadCount('getBlocks')

33
onionr/communicatorutils/downloadblocks/shoulddownload.py Normal file
View File

@ -0,0 +1,33 @@
'''
Onionr - Private P2P Communication
Check if a block should be downloaded (if we already have it or its blacklisted or not)
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
def should_download(comm_inst, block_hash):
ret_data = True
if block_hash in comm_inst._core.getBlockList(): # Dont download block we have
ret_data = False
else:
if comm_inst._core._blacklist.inBlacklist(block_hash): # Dont download blacklisted block
ret_data = False
if ret_data is False:
# Remove block from communicator queue if it shouldnt be downloaded
try:
del comm_inst.blockQueue[block_hash]
except KeyError:
pass
return ret_data

60
onionr/communicatorutils/housekeeping.py Executable file
View File

@ -0,0 +1,60 @@
'''
Onionr - Private P2P Communication
Cleanup old Onionr blocks and forward secrecy keys using the communicator. Ran from a timer usually
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3
import logger
from onionrusers import onionrusers
from onionrutils import epoch
def clean_old_blocks(comm_inst):
'''Delete old blocks if our disk allocation is full/near full, and also expired blocks'''
# Delete expired blocks
for bHash in comm_inst._core.getExpiredBlocks():
comm_inst._core._blacklist.addToDB(bHash)
comm_inst._core.removeBlock(bHash)
logger.info('Deleted block: %s' % (bHash,))
while comm_inst._core.storage_counter.isFull():
oldest = comm_inst._core.getBlockList()[0]
comm_inst._core._blacklist.addToDB(oldest)
comm_inst._core.removeBlock(oldest)
logger.info('Deleted block: %s' % (oldest,))
comm_inst.decrementThreadCount('clean_old_blocks')
def clean_keys(comm_inst):
'''Delete expired forward secrecy keys'''
conn = sqlite3.connect(comm_inst._core.peerDB, timeout=10)
c = conn.cursor()
time = epoch.get_epoch()
deleteKeys = []
for entry in c.execute("SELECT * FROM forwardKeys WHERE expire <= ?", (time,)):
logger.debug('Forward key: %s' % entry[1])
deleteKeys.append(entry[1])
for key in deleteKeys:
logger.debug('Deleting forward key %s' % key)
c.execute("DELETE from forwardKeys where forwardKey = ?", (key,))
conn.commit()
conn.close()
onionrusers.deleteExpiredKeys(comm_inst._core)
comm_inst.decrementThreadCount('clean_keys')

49
onionr/communicatorutils/lookupadders.py Executable file
View File

@ -0,0 +1,49 @@
'''
Onionr - Private P2P Communication
Lookup new peer transport addresses using the communicator
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import logger
from onionrutils import stringvalidators
from communicator import peeraction, onlinepeers
def lookup_new_peer_transports_with_communicator(comm_inst):
logger.info('Looking up new addresses...')
tryAmount = 1
newPeers = []
for i in range(tryAmount):
# Download new peer address list from random online peers
if len(newPeers) > 10000:
# Don't get new peers if we have too many queued up
break
peer = onlinepeers.pick_online_peer(comm_inst)
newAdders = peeraction.peer_action(comm_inst, peer, action='pex')
try:
newPeers = newAdders.split(',')
except AttributeError:
pass
else:
# Validate new peers are good format and not already in queue
invalid = []
for x in newPeers:
x = x.strip()
if not stringvalidators.validate_transport(x) or x in comm_inst.newPeers or x == comm_inst._core.hsAddress:
# avoid adding if its our address
invalid.append(x)
for x in invalid:
newPeers.remove(x)
comm_inst.newPeers.extend(newPeers)
comm_inst.decrementThreadCount('lookupAdders')

89
onionr/communicatorutils/lookupblocks.py Executable file
View File

@ -0,0 +1,89 @@
'''
Onionr - Private P2P Communication
Lookup new blocks with the communicator using a random connected peer
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import logger, onionrproofs
from onionrutils import stringvalidators, epoch
from communicator import peeraction, onlinepeers
def lookup_blocks_from_communicator(comm_inst):
logger.info('Looking up new blocks...')
tryAmount = 2
newBlocks = ''
existingBlocks = comm_inst._core.getBlockList()
triedPeers = [] # list of peers we've tried this time around
maxBacklog = 1560 # Max amount of *new* block hashes to have already in queue, to avoid memory exhaustion
lastLookupTime = 0 # Last time we looked up a particular peer's list
new_block_count = 0
for i in range(tryAmount):
listLookupCommand = 'getblocklist' # This is defined here to reset it each time
if len(comm_inst.blockQueue) >= maxBacklog:
break
if not comm_inst.isOnline:
break
# check if disk allocation is used
if comm_inst._core.storage_counter.isFull():
logger.debug('Not looking up new blocks due to maximum amount of allowed disk space used')
break
peer = onlinepeers.pick_online_peer(comm_inst) # select random online peer
# if we've already tried all the online peers this time around, stop
if peer in triedPeers:
if len(comm_inst.onlinePeers) == len(triedPeers):
break
else:
continue
triedPeers.append(peer)
# Get the last time we looked up a peer's stamp to only fetch blocks since then.
# Saved in memory only for privacy reasons
try:
lastLookupTime = comm_inst.dbTimestamps[peer]
except KeyError:
lastLookupTime = 0
else:
listLookupCommand += '?date=%s' % (lastLookupTime,)
try:
newBlocks = peeraction.peer_action(comm_inst, peer, listLookupCommand) # get list of new block hashes
except Exception as error:
logger.warn('Could not get new blocks from %s.' % peer, error = error)
newBlocks = False
else:
comm_inst.dbTimestamps[peer] = epoch.get_rounded_epoch(roundS=60)
if newBlocks != False:
# if request was a success
for i in newBlocks.split('\n'):
if stringvalidators.validate_hash(i):
# if newline seperated string is valid hash
if not i in existingBlocks:
# if block does not exist on disk and is not already in block queue
if i not in comm_inst.blockQueue:
if onionrproofs.hashMeetsDifficulty(i) and not comm_inst._core._blacklist.inBlacklist(i):
if len(comm_inst.blockQueue) <= 1000000:
comm_inst.blockQueue[i] = [peer] # add blocks to download queue
new_block_count += 1
else:
if peer not in comm_inst.blockQueue[i]:
if len(comm_inst.blockQueue[i]) < 10:
comm_inst.blockQueue[i].append(peer)
if new_block_count > 0:
block_string = ""
if new_block_count > 1:
block_string = "s"
logger.info('Discovered %s new block%s' % (new_block_count, block_string), terminal=True)
comm_inst.decrementThreadCount('lookupBlocks')
return

41
onionr/communicatorutils/netcheck.py Executable file
View File

@ -0,0 +1,41 @@
'''
Onionr - Private P2P Communication
Determine if our node is able to use Tor based on the status of a communicator instance
and the result of pinging onion http servers
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import logger
from utils import netutils
from onionrutils import localcommand, epoch
def net_check(comm_inst):
'''Check if we are connected to the internet or not when we can't connect to any peers'''
rec = False # for detecting if we have received incoming connections recently
c = comm_inst._core
if len(comm_inst.onlinePeers) == 0:
try:
if (epoch.get_epoch() - int(localcommand.local_command(c, '/lastconnect'))) <= 60:
comm_inst.isOnline = True
rec = True
except ValueError:
pass
if not rec and not netutils.checkNetwork(c, torPort=comm_inst.proxyPort):
if not comm_inst.shutdown:
logger.warn('Network check failed, are you connected to the Internet, and is Tor working?')
comm_inst.isOnline = False
else:
comm_inst.isOnline = True
comm_inst.decrementThreadCount('net_check')

10
onionr/communicatorutils/onionrcommunicatortimers.py
View File

@ -1,6 +1,5 @@
#!/usr/bin/env python3
'''
Onionr - P2P Anonymous Storage Network
Onionr - Private P2P Communication
This file contains timer control for the communicator
'''
@ -20,7 +19,7 @@
'''
import threading, onionrexceptions, logger
class OnionrCommunicatorTimers:
def __init__(self, daemonInstance, timerFunction, frequency, makeThread=True, threadAmount=1, maxThreads=5, requiresPeer=False):
def __init__(self, daemonInstance, timerFunction, frequency, makeThread=True, threadAmount=1, maxThreads=5, requiresPeer=False, myArgs=[]):
self.timerFunction = timerFunction
self.frequency = frequency
self.threadAmount = threadAmount
@ -29,6 +28,7 @@ class OnionrCommunicatorTimers:
self.daemonInstance = daemonInstance
self.maxThreads = maxThreads
self._core = self.daemonInstance._core
self.args = myArgs
self.daemonInstance.timers.append(self)
self.count = 0
@ -52,10 +52,10 @@ class OnionrCommunicatorTimers:
if self.makeThread:
for i in range(self.threadAmount):
if self.daemonInstance.threadCounts[self.timerFunction.__name__] >= self.maxThreads:
logger.debug('%s is currently using the maximum number of threads, not starting another.' % self.timerFunction.__name__)
logger.debug('%s is currently using the maximum number of threads, not starting another.' % self.timerFunction.__name__, terminal=True)
else:
self.daemonInstance.threadCounts[self.timerFunction.__name__] += 1
newThread = threading.Thread(target=self.timerFunction)
newThread = threading.Thread(target=self.timerFunction, args=self.args, daemon=True)
newThread.start()
else:
self.timerFunction()

214
onionr/communicatorutils/onionrdaemontools.py
View File

@ -1,214 +0,0 @@
'''
Onionr - P2P Anonymous Storage Network
Contains the CommunicatorUtils class which contains useful functions for the communicator daemon
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import onionrexceptions, onionrpeers, onionrproofs, logger
import base64, sqlite3, os
from dependencies import secrets
from utils import netutils
from onionrusers import onionrusers
class DaemonTools:
'''
Class intended for use by Onionr Communicator
'''
def __init__(self, daemon):
self.daemon = daemon
self.announceProgress = {}
self.announceCache = {}
def announceNode(self):
'''Announce our node to our peers'''
retData = False
announceFail = False
# Do not let announceCache get too large
if len(self.announceCache) >= 10000:
self.announceCache.popitem()
if self.daemon._core.config.get('general.security_level', 0) == 0:
# Announce to random online peers
for i in self.daemon.onlinePeers:
if not i in self.announceCache and not i in self.announceProgress:
peer = i
break
else:
peer = self.daemon.pickOnlinePeer()
for x in range(1):
if x == 1 and self.daemon._core.config.get('i2p.host'):
ourID = self.daemon._core.config.get('i2p.own_addr').strip()
else:
ourID = self.daemon._core.hsAddress.strip()
url = 'http://' + peer + '/announce'
data = {'node': ourID}
combinedNodes = ourID + peer
if ourID != 1:
#TODO: Extend existingRand for i2p
existingRand = self.daemon._core.getAddressInfo(peer, 'powValue')
if type(existingRand) is type(None):
existingRand = ''
if peer in self.announceCache:
data['random'] = self.announceCache[peer]
elif len(existingRand) > 0:
data['random'] = existingRand
else:
self.announceProgress[peer] = True
proof = onionrproofs.DataPOW(combinedNodes, forceDifficulty=4)
del self.announceProgress[peer]
try:
data['random'] = base64.b64encode(proof.waitForResult()[1])
except TypeError:
# Happens when we failed to produce a proof
logger.error("Failed to produce a pow for announcing to " + peer)
announceFail = True
else:
self.announceCache[peer] = data['random']
if not announceFail:
logger.info('Announcing node to ' + url)
if self.daemon._core._utils.doPostRequest(url, data) == 'Success':
logger.info('Successfully introduced node to ' + peer)
retData = True
self.daemon._core.setAddressInfo(peer, 'introduced', 1)
self.daemon._core.setAddressInfo(peer, 'powValue', data['random'])
self.daemon.decrementThreadCount('announceNode')
return retData
def netCheck(self):
'''Check if we are connected to the internet or not when we can't connect to any peers'''
if len(self.daemon.onlinePeers) == 0:
if not netutils.checkNetwork(self.daemon._core._utils, torPort=self.daemon.proxyPort):
if not self.daemon.shutdown:
logger.warn('Network check failed, are you connected to the internet?')
self.daemon.isOnline = False
else:
self.daemon.isOnline = True
self.daemon.decrementThreadCount('netCheck')
def cleanOldBlocks(self):
'''Delete old blocks if our disk allocation is full/near full, and also expired blocks'''
# Delete expired blocks
for bHash in self.daemon._core.getExpiredBlocks():
self.daemon._core._blacklist.addToDB(bHash)
self.daemon._core.removeBlock(bHash)
logger.info('Deleted block: %s' % (bHash,))
while self.daemon._core._utils.storageCounter.isFull():
oldest = self.daemon._core.getBlockList()[0]
self.daemon._core._blacklist.addToDB(oldest)
self.daemon._core.removeBlock(oldest)
logger.info('Deleted block: %s' % (oldest,))
self.daemon.decrementThreadCount('cleanOldBlocks')
def cleanKeys(self):
'''Delete expired forward secrecy keys'''
conn = sqlite3.connect(self.daemon._core.peerDB, timeout=10)
c = conn.cursor()
time = self.daemon._core._utils.getEpoch()
deleteKeys = []
for entry in c.execute("SELECT * FROM forwardKeys WHERE expire <= ?", (time,)):
logger.debug('Forward key: %s' % entry[1])
deleteKeys.append(entry[1])
for key in deleteKeys:
logger.debug('Deleting forward key %s' % key)
c.execute("DELETE from forwardKeys where forwardKey = ?", (key,))
conn.commit()
conn.close()
onionrusers.deleteExpiredKeys(self.daemon._core)
self.daemon.decrementThreadCount('cleanKeys')
def cooldownPeer(self):
'''Randomly add an online peer to cooldown, so we can connect a new one'''
onlinePeerAmount = len(self.daemon.onlinePeers)
minTime = 300
cooldownTime = 600
toCool = ''
tempConnectTimes = dict(self.daemon.connectTimes)
# Remove peers from cooldown that have been there long enough
tempCooldown = dict(self.daemon.cooldownPeer)
for peer in tempCooldown:
if (self.daemon._core._utils.getEpoch() - tempCooldown[peer]) >= cooldownTime:
del self.daemon.cooldownPeer[peer]
# Cool down a peer, if we have max connections alive for long enough
if onlinePeerAmount >= self.daemon._core.config.get('peers.max_connect', 10, save = True):
finding = True
while finding:
try:
toCool = min(tempConnectTimes, key=tempConnectTimes.get)
if (self.daemon._core._utils.getEpoch() - tempConnectTimes[toCool]) < minTime:
del tempConnectTimes[toCool]
else:
finding = False
except ValueError:
break
else:
self.daemon.removeOnlinePeer(toCool)
self.daemon.cooldownPeer[toCool] = self.daemon._core._utils.getEpoch()
self.daemon.decrementThreadCount('cooldownPeer')
def runCheck(self):
if os.path.isfile(self.daemon._core.dataDir + '.runcheck'):
os.remove(self.daemon._core.dataDir + '.runcheck')
return True
return False
def humanReadableTime(self, seconds):
build = ''
units = {
'year' : 31557600,
'month' : (31557600 / 12),
'day' : 86400,
'hour' : 3600,
'minute' : 60,
'second' : 1
}
for unit in units:
amnt_unit = int(seconds / units[unit])
if amnt_unit >= 1:
seconds -= amnt_unit * units[unit]
build += '%s %s' % (amnt_unit, unit) + ('s' if amnt_unit != 1 else '') + ' '
return build.strip()
def insertDeniableBlock(self):
'''Insert a fake block in order to make it more difficult to track real blocks'''
fakePeer = ''
chance = 10
if secrets.randbelow(chance) == (chance - 1):
fakePeer = 'OVPCZLOXD6DC5JHX4EQ3PSOGAZ3T24F75HQLIUZSDSMYPEOXCPFA===='
data = secrets.token_hex(secrets.randbelow(500) + 1)
self.daemon._core.insertBlock(data, header='pm', encryptType='asym', asymPeer=fakePeer, meta={'subject': 'foo'})
self.daemon.decrementThreadCount('insertDeniableBlock')
return

5
onionr/communicatorutils/proxypicker.py Normal file → Executable file
View File

@ -1,7 +1,7 @@
'''
Onionr - P2P Anonymous Storage Network
Onionr - Private P2P Communication
Just picks a proxy
Just picks a proxy to use based on a peer's address
'''
'''
This program is free software: you can redistribute it and/or modify
@ -23,3 +23,4 @@ def pick_proxy(peer_address):
return 'tor'
elif peer_address.endswith('.i2p'):
return 'i2p'
raise ValueError("Peer address was not string ending with acceptable value")

40
onionr/communicatorutils/servicecreator.py Executable file
View File

@ -0,0 +1,40 @@
'''
Onionr - Private P2P Communication
Creates an onionr direct connection service by scanning all connection blocks
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import communicator, onionrblockapi
from onionrutils import stringvalidators, bytesconverter
def service_creator(daemon):
assert isinstance(daemon, communicator.OnionrCommunicatorDaemon)
core = daemon._core
# Find socket connection blocks
# TODO cache blocks and only look at recently received ones
con_blocks = core.getBlocksByType('con')
for b in con_blocks:
if not b in daemon.active_services:
bl = onionrblockapi.Block(b, core=core, decrypt=True)
bs = bytesconverter.bytes_to_str(bl.bcontent) + '.onion'
if stringvalidators.validate_pub_key(bl.signer) and stringvalidators.validate_transport(bs):
signer = bytesconverter.bytes_to_str(bl.signer)
daemon.active_services.append(b)
daemon.active_services.append(signer)
daemon.services.create_server(signer, bs)
daemon.decrementThreadCount('service_creator')

57
onionr/communicatorutils/uploadblocks.py Executable file
View File

@ -0,0 +1,57 @@
'''
Onionr - Private P2P Communication
Upload blocks in the upload queue to peers from the communicator
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import logger
from communicatorutils import proxypicker
import onionrblockapi as block
from onionrutils import localcommand, stringvalidators, basicrequests
from communicator import onlinepeers
def upload_blocks_from_communicator(comm_inst):
# when inserting a block, we try to upload it to a few peers to add some deniability
TIMER_NAME = "upload_blocks_from_communicator"
triedPeers = []
finishedUploads = []
core = comm_inst._core
comm_inst.blocksToUpload = core._crypto.randomShuffle(comm_inst.blocksToUpload)
if len(comm_inst.blocksToUpload) != 0:
for bl in comm_inst.blocksToUpload:
if not stringvalidators.validate_hash(bl):
logger.warn('Requested to upload invalid block', terminal=True)
comm_inst.decrementThreadCount(TIMER_NAME)
return
for i in range(min(len(comm_inst.onlinePeers), 6)):
peer = onlinepeers.pick_online_peer(comm_inst)
if peer in triedPeers:
continue
triedPeers.append(peer)
url = 'http://' + peer + '/upload'
data = {'block': block.Block(bl).getRaw()}
proxyType = proxypicker.pick_proxy(peer)
logger.info("Uploading block to " + peer, terminal=True)
if not basicrequests.do_post_request(core, url, data=data, proxyType=proxyType) == False:
localcommand.local_command(core, 'waitforshare/' + bl, post=True)
finishedUploads.append(bl)
for x in finishedUploads:
try:
comm_inst.blocksToUpload.remove(x)
except ValueError:
pass
comm_inst.decrementThreadCount(TIMER_NAME)

20
onionr/config.py
View File

@ -1,5 +1,5 @@
'''
Onionr - P2P Microblogging Platform & Social network
Onionr - Private P2P Communication
This file deals with configuration management.
'''
@ -19,11 +19,9 @@
'''
import os, json, logger
from utils import identifyhome
# set data dir
dataDir = os.environ.get('ONIONR_HOME', os.environ.get('DATA_DIR', 'data/'))
if not dataDir.endswith('/'):
dataDir += '/'
dataDir = identifyhome.identify_home()
_configfile = os.path.abspath(dataDir + 'config.json')
_config = {}
@ -96,15 +94,8 @@ def check():
Checks if the configuration file exists, creates it if not
'''
try:
if not os.path.exists(os.path.dirname(get_config_file())):
os.path.mkdirs(os.path.dirname(get_config_file()))
if not os.path.isfile(get_config_file()):
open(get_config_file(), 'a', encoding="utf8").close()
save()
except:
pass
#logger.debug('Failed to check configuration file.')
os.makedirs(os.path.dirname(get_config_file()))
def save():
'''
@ -115,14 +106,13 @@ def save():
try:
with open(get_config_file(), 'w', encoding="utf8") as configfile:
json.dump(get_config(), configfile, indent=2)
except:
except json.JSONDecodeError:
logger.warn('Failed to write to configuration file.')
def reload():
'''
Reloads the configuration data in memory from the file
'''
check()
try:
with open(get_config_file(), 'r', encoding="utf8") as configfile:

527
onionr/core.py
View File

@ -1,5 +1,5 @@
'''
Onionr - P2P Anonymous Storage Network
Onionr - Private P2P Communication
Core Onionr library, useful for external programs. Handles peer & data processing
'''
@ -17,34 +17,35 @@
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3, os, sys, time, math, base64, tarfile, nacl, logger, json, netcontroller, math, config, uuid
import os, sys, json
import logger, netcontroller, config
from onionrblockapi import Block
import coredb
import deadsimplekv as simplekv
import onionrutils, onionrcrypto, onionrproofs, onionrevents as events, onionrexceptions
import onionrcrypto, onionrproofs, onionrevents as events, onionrexceptions
import onionrblacklist
from onionrusers import onionrusers
from onionrstorage import removeblock, setdata
import dbcreator, onionrstorage, serializeddata, subprocesspow
from etc import onionrvalues
if sys.version_info < (3, 6):
try:
import sha3
except ModuleNotFoundError:
logger.fatal('On Python 3 versions prior to 3.6.x, you need the sha3 module')
sys.exit(1)
from etc import onionrvalues, powchoice
from onionrutils import localcommand, stringvalidators, bytesconverter, epoch
from onionrutils import blockmetadata
from utils import identifyhome
import storagecounter
class Core:
def __init__(self, torPort=0):
'''
Initialize Core Onionr library
'''
# set data dir
self.dataDir = os.environ.get('ONIONR_HOME', os.environ.get('DATA_DIR', 'data/'))
if not self.dataDir.endswith('/'):
self.dataDir += '/'
self.dataDir = identifyhome.identify_home()
try:
self.usageFile = self.dataDir + 'disk-usage.txt'
self.config = config
self.maxBlockSize = 10000000 # max block size in bytes
self.onionrInst = None
self.queueDB = self.dataDir + 'queue.db'
self.peerDB = self.dataDir + 'peers.db'
@ -64,6 +65,7 @@ class Core:
self.dbCreate = dbcreator.DBCreator(self)
self.forwardKeysFile = self.dataDir + 'forward-keys.db'
self.keyStore = simplekv.DeadSimpleKV(self.dataDir + 'cachedstorage.dat', refresh_seconds=5)
self.storage_counter = storagecounter.StorageCounter(self)
# Socket data, defined here because of multithreading constraints with gevent
self.killSockets = False
@ -72,11 +74,6 @@ class Core:
self.socketReasons = {}
self.socketServerResponseData = {}
self.usageFile = self.dataDir + 'disk-usage.txt'
self.config = config
self.maxBlockSize = 10000000 # max block size in bytes
if not os.path.exists(self.dataDir):
os.mkdir(self.dataDir)
if not os.path.exists(self.dataDir + 'blocks/'):
@ -103,15 +100,15 @@ class Core:
else:
logger.warn('Warning: address bootstrap file not found ' + self.bootstrapFileLocation)
self._utils = onionrutils.OnionrUtils(self)
self.use_subprocess = powchoice.use_subprocess(self)
# Initialize the crypto object
self._crypto = onionrcrypto.OnionrCrypto(self)
self._blacklist = onionrblacklist.OnionrBlackList(self)
self.serializer = serializeddata.SerializedData(self)
except Exception as error:
logger.error('Failed to initialize core Onionr library.', error=error)
logger.fatal('Cannot recover from error.')
logger.error('Failed to initialize core Onionr library.', error=error, terminal=True)
logger.fatal('Cannot recover from error.', terminal=True)
sys.exit(1)
return
@ -127,88 +124,19 @@ class Core:
'''
Adds a public key to the key database (misleading function name)
'''
assert peerID not in self.listPeers()
# This function simply adds a peer to the DB
if not self._utils.validatePubKey(peerID):
return False
events.event('pubkey_add', data = {'key': peerID}, onionr = None)
conn = sqlite3.connect(self.peerDB, timeout=30)
hashID = self._crypto.pubKeyHashID(peerID)
c = conn.cursor()
t = (peerID, name, 'unknown', hashID, 0)
for i in c.execute("SELECT * FROM peers WHERE id = ?;", (peerID,)):
try:
if i[0] == peerID:
conn.close()
return False
except ValueError:
pass
except IndexError:
pass
c.execute('INSERT INTO peers (id, name, dateSeen, hashID, trust) VALUES(?, ?, ?, ?, ?);', t)
conn.commit()
conn.close()
return True
return coredb.keydb.addkeys.add_peer(self, peerID, name)
def addAddress(self, address):
'''
Add an address to the address database (only tor currently)
'''
if address == config.get('i2p.ownAddr', None) or address == self.hsAddress:
return False
if type(address) is None or len(address) == 0:
return False
if self._utils.validateID(address):
conn = sqlite3.connect(self.addressDB, timeout=30)
c = conn.cursor()
# check if address is in database
# this is safe to do because the address is validated above, but we strip some chars here too just in case
address = address.replace('\'', '').replace(';', '').replace('"', '').replace('\\', '')
for i in c.execute("SELECT * FROM adders WHERE address = ?;", (address,)):
try:
if i[0] == address:
conn.close()
return False
except ValueError:
pass
except IndexError:
pass
t = (address, 1)
c.execute('INSERT INTO adders (address, type) VALUES(?, ?);', t)
conn.commit()
conn.close()
events.event('address_add', data = {'address': address}, onionr = None)
return True
else:
#logger.debug('Invalid ID: %s' % address)
return False
return coredb.keydb.addkeys.add_address(self, address)
def removeAddress(self, address):
'''
Remove an address from the address database
'''
if self._utils.validateID(address):
conn = sqlite3.connect(self.addressDB, timeout=30)
c = conn.cursor()
t = (address,)
c.execute('Delete from adders where address=?;', t)
conn.commit()
conn.close()
events.event('address_remove', data = {'address': address}, onionr = None)
return True
else:
return False
return coredb.keydb.removekeys.remove_address(self, address)
def removeBlock(self, block):
'''
@ -216,16 +144,7 @@ class Core:
**You may want blacklist.addToDB(blockHash)
'''
if self._utils.validateHash(block):
conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor()
t = (block,)
c.execute('Delete from hashes where hash=?;', t)
conn.commit()
conn.close()
dataSize = sys.getsizeof(onionrstorage.getData(self, block))
self._utils.storageCounter.removeBytes(dataSize)
removeblock.remove_block(self, block)
def createAddressDB(self):
'''
@ -251,67 +170,19 @@ class Core:
Should be in hex format!
'''
if not os.path.exists(self.blockDB):
raise Exception('Block db does not exist')
if self._utils.hasBlock(newHash):
return
conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor()
currentTime = self._utils.getEpoch() + self._crypto.secrets.randbelow(301)
if selfInsert or dataSaved:
selfInsert = 1
else:
selfInsert = 0
data = (newHash, currentTime, '', selfInsert)
c.execute('INSERT INTO hashes (hash, dateReceived, dataType, dataSaved) VALUES(?, ?, ?, ?);', data)
conn.commit()
conn.close()
return
def getData(self, hash):
'''
Simply return the data associated to a hash
'''
data = onionrstorage.getData(self, hash)
return data
coredb.blockmetadb.add.add_to_block_DB(self, newHash, selfInsert, dataSaved)
def setData(self, data):
'''
Set the data assciated with a hash
'''
return onionrstorage.setdata.set_data(self, data)
data = data
dataSize = sys.getsizeof(data)
if not type(data) is bytes:
data = data.encode()
dataHash = self._crypto.sha3Hash(data)
if type(dataHash) is bytes:
dataHash = dataHash.decode()
blockFileName = self.blockDataLocation + dataHash + '.dat'
if os.path.exists(blockFileName):
pass # TODO: properly check if block is already saved elsewhere
#raise Exception("Data is already set for " + dataHash)
else:
if self._utils.storageCounter.addBytes(dataSize) != False:
onionrstorage.store(self, data, blockHash=dataHash)
conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor()
c.execute("UPDATE hashes SET dataSaved=1 WHERE hash = ?;", (dataHash,))
conn.commit()
conn.close()
with open(self.dataNonceFile, 'a') as nonceFile:
nonceFile.write(dataHash + '\n')
else:
raise onionrexceptions.DiskAllocationReached
return dataHash
def getData(self, hash):
'''
Simply return the data associated to a hash
'''
return onionrstorage.getData(self, hash)
def daemonQueue(self):
'''
@ -319,118 +190,31 @@ class Core:
This function intended to be used by the client. Queue to exchange data between "client" and server.
'''
retData = False
if not os.path.exists(self.queueDB):
self.dbCreate.createDaemonDB()
else:
conn = sqlite3.connect(self.queueDB, timeout=30)
c = conn.cursor()
try:
for row in c.execute('SELECT command, data, date, min(ID), responseID FROM commands group by id'):
retData = row
break
except sqlite3.OperationalError:
self.dbCreate.createDaemonDB()
else:
if retData != False:
c.execute('DELETE FROM commands WHERE id=?;', (retData[3],))
conn.commit()
conn.close()
events.event('queue_pop', data = {'data': retData}, onionr = None)
return retData
return coredb.daemonqueue.daemon_queue(self)
def daemonQueueAdd(self, command, data='', responseID=''):
'''
Add a command to the daemon queue, used by the communication daemon (communicator.py)
'''
retData = True
date = self._utils.getEpoch()
conn = sqlite3.connect(self.queueDB, timeout=30)
c = conn.cursor()
t = (command, data, date, responseID)
try:
c.execute('INSERT INTO commands (command, data, date, responseID) VALUES(?, ?, ?, ?)', t)
conn.commit()
except sqlite3.OperationalError:
retData = False
self.daemonQueue()
events.event('queue_push', data = {'command': command, 'data': data}, onionr = None)
conn.close()
return retData
return coredb.daemonqueue.daemon_queue_add(self, command, data, responseID)
def daemonQueueGetResponse(self, responseID=''):
'''
Get a response sent by communicator to the API, by requesting to the API
'''
assert len(responseID) > 0
resp = self._utils.localCommand('queueResponse/' + responseID)
return resp
def daemonQueueWaitForResponse(self, responseID='', checkFreqSecs=1):
resp = 'failure'
while resp == 'failure':
resp = self.daemonQueueGetResponse(responseID)
time.sleep(1)
return resp
def daemonQueueSimple(self, command, data='', checkFreqSecs=1):
'''
A simplified way to use the daemon queue. Will register a command (with optional data) and wait, return the data
Not always useful, but saves time + LOC in some cases.
This is a blocking function, so be careful.
'''
responseID = str(uuid.uuid4()) # generate unique response ID
self.daemonQueueAdd(command, data=data, responseID=responseID)
return self.daemonQueueWaitForResponse(responseID, checkFreqSecs)
return coredb.daemonqueue.daemon_queue_get_response(self, responseID)
def clearDaemonQueue(self):
'''
Clear the daemon queue (somewhat dangerous)
'''
conn = sqlite3.connect(self.queueDB, timeout=30)
c = conn.cursor()
try:
c.execute('DELETE FROM commands;')
conn.commit()
except:
pass
conn.close()
events.event('queue_clear', onionr = None)
return
return coredb.daemonqueue.clear_daemon_queue(self)
def listAdders(self, randomOrder=True, i2p=True, recent=0):
'''
Return a list of addresses
'''
conn = sqlite3.connect(self.addressDB, timeout=30)
c = conn.cursor()
if randomOrder:
addresses = c.execute('SELECT * FROM adders ORDER BY RANDOM();')
else:
addresses = c.execute('SELECT * FROM adders;')
addressList = []
for i in addresses:
if len(i[0].strip()) == 0:
continue
addressList.append(i[0])
conn.close()
testList = list(addressList) # create new list to iterate
for address in testList:
try:
if recent > 0 and (self._utils.getEpoch() - self.getAddressInfo(address, 'lastConnect')) > recent:
raise TypeError # If there is no last-connected date or it was too long ago, don't add peer to list if recent is not 0
except TypeError:
addressList.remove(address)
return addressList
return coredb.keydb.listkeys.list_adders(self, randomOrder, i2p, recent)
def listPeers(self, randomOrder=True, getPow=False, trust=0):
'''
@ -439,35 +223,7 @@ class Core:
randomOrder determines if the list should be in a random order
trust sets the minimum trust to list
'''
conn = sqlite3.connect(self.peerDB, timeout=30)
c = conn.cursor()
payload = ''
if trust not in (0, 1, 2):
logger.error('Tried to select invalid trust.')
return
if randomOrder:
payload = 'SELECT * FROM peers WHERE trust >= ? ORDER BY RANDOM();'
else:
payload = 'SELECT * FROM peers WHERE trust >= ?;'
peerList = []
for i in c.execute(payload, (trust,)):
try:
if len(i[0]) != 0:
if getPow:
peerList.append(i[0] + '-' + i[1])
else:
peerList.append(i[0])
except TypeError:
pass
conn.close()
return peerList
return coredb.keydb.listkeys.list_peers(self, randomOrder, getPow, trust)
def getPeerInfo(self, peer, info):
'''
@ -480,46 +236,13 @@ class Core:
trust int 4
hashID text 5
'''
conn = sqlite3.connect(self.peerDB, timeout=30)
c = conn.cursor()
command = (peer,)
infoNumbers = {'id': 0, 'name': 1, 'adders': 2, 'dateSeen': 3, 'trust': 4, 'hashID': 5}
info = infoNumbers[info]
iterCount = 0
retVal = ''
for row in c.execute('SELECT * FROM peers WHERE id=?;', command):
for i in row:
if iterCount == info:
retVal = i
break
else:
iterCount += 1
conn.close()
return retVal
return coredb.keydb.userinfo.get_user_info(self, peer, info)
def setPeerInfo(self, peer, key, data):
'''
Update a peer for a key
'''
conn = sqlite3.connect(self.peerDB, timeout=30)
c = conn.cursor()
command = (data, peer)
# TODO: validate key on whitelist
if key not in ('id', 'name', 'pubkey', 'forwardKey', 'dateSeen', 'trust'):
raise Exception("Got invalid database key when setting peer info")
c.execute('UPDATE peers SET ' + key + ' = ? WHERE id=?', command)
conn.commit()
conn.close()
return
return coredb.keydb.userinfo.set_peer_info(self, peer, key, data)
def getAddressInfo(self, address, info):
'''
@ -536,117 +259,35 @@ class Core:
trust 8
introduced 9
'''
conn = sqlite3.connect(self.addressDB, timeout=30)
c = conn.cursor()
command = (address,)
infoNumbers = {'address': 0, 'type': 1, 'knownPeer': 2, 'speed': 3, 'success': 4, 'powValue': 5, 'failure': 6, 'lastConnect': 7, 'trust': 8, 'introduced': 9}
info = infoNumbers[info]
iterCount = 0
retVal = ''
for row in c.execute('SELECT * FROM adders WHERE address=?;', command):
for i in row:
if iterCount == info:
retVal = i
break
else:
iterCount += 1
conn.close()
return retVal
return coredb.keydb.transportinfo.get_address_info(self, address, info)
def setAddressInfo(self, address, key, data):
'''
Update an address for a key
'''
conn = sqlite3.connect(self.addressDB, timeout=30)
c = conn.cursor()
command = (data, address)
if key not in ('address', 'type', 'knownPeer', 'speed', 'success', 'failure', 'powValue', 'lastConnect', 'lastConnectAttempt', 'trust', 'introduced'):
raise Exception("Got invalid database key when setting address info")
else:
c.execute('UPDATE adders SET ' + key + ' = ? WHERE address=?', command)
conn.commit()
conn.close()
return
return coredb.keydb.transportinfo.set_address_info(self, address, key, data)
def getBlockList(self, dateRec = None, unsaved = False):
'''
Get list of our blocks
'''
if dateRec == None:
dateRec = 0
conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor()
execute = 'SELECT hash FROM hashes WHERE dateReceived >= ? ORDER BY dateReceived ASC;'
args = (dateRec,)
rows = list()
for row in c.execute(execute, args):
for i in row:
rows.append(i)
conn.close()
return rows
return coredb.blockmetadb.get_block_list(self, dateRec, unsaved)
def getBlockDate(self, blockHash):
'''
Returns the date a block was received
'''
conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor()
execute = 'SELECT dateReceived FROM hashes WHERE hash=?;'
args = (blockHash,)
for row in c.execute(execute, args):
for i in row:
return int(i)
conn.close()
return None
return coredb.blockmetadb.get_block_date(self, blockHash)
def getBlocksByType(self, blockType, orderDate=True):
'''
Returns a list of blocks by the type
'''
conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor()
if orderDate:
execute = 'SELECT hash FROM hashes WHERE dataType=? ORDER BY dateReceived;'
else:
execute = 'SELECT hash FROM hashes WHERE dataType=?;'
args = (blockType,)
rows = list()
for row in c.execute(execute, args):
for i in row:
rows.append(i)
conn.close()
return rows
return coredb.blockmetadb.get_blocks_by_type(self, blockType, orderDate)
def getExpiredBlocks(self):
'''Returns a list of expired blocks'''
conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor()
date = int(self._utils.getEpoch())
execute = 'SELECT hash FROM hashes WHERE expire <= %s ORDER BY dateReceived;' % (date,)
rows = list()
for row in c.execute(execute):
for i in row:
rows.append(i)
conn.close()
return rows
return coredb.blockmetadb.expiredblocks.get_expired_blocks(self)
def updateBlockInfo(self, hash, key, data):
'''
@ -663,18 +304,7 @@ class Core:
dateClaimed - timestamp claimed inside the block, only as trustworthy as the block author is
expire - expire date for a block
'''
if key not in ('dateReceived', 'decrypted', 'dataType', 'dataFound', 'dataSaved', 'sig', 'author', 'dateClaimed', 'expire'):
return False
conn = sqlite3.connect(self.blockDB, timeout=30)
c = conn.cursor()
args = (data, hash)
c.execute("UPDATE hashes SET " + key + " = ? where hash = ?;", args)
conn.commit()
conn.close()
return True
return coredb.blockmetadb.updateblockinfo.update_block_info(self, hash, key, data)
def insertBlock(self, data, header='txt', sign=False, encryptType='', symKey='', asymPeer='', meta = {}, expire=None, disableForward=False):
'''
@ -682,15 +312,17 @@ class Core:
encryptType must be specified to encrypt a block
'''
allocationReachedMessage = 'Cannot insert block, disk allocation reached.'
if self._utils.storageCounter.isFull():
if self.storage_counter.isFull():
logger.error(allocationReachedMessage)
return False
retData = False
createTime = self._utils.getRoundedEpoch()
if type(data) is None:
raise ValueError('Data cannot be none')
# check nonce
dataNonce = self._utils.bytesToStr(self._crypto.sha3Hash(data))
createTime = epoch.get_epoch()
dataNonce = bytesconverter.bytes_to_str(self._crypto.sha3Hash(data))
try:
with open(self.dataNonceFile, 'r') as nonces:
if dataNonce in nonces:
@ -763,14 +395,18 @@ class Core:
signature = self._crypto.symmetricEncrypt(signature, key=symKey, returnEncoded=True).decode()
signer = self._crypto.symmetricEncrypt(signer, key=symKey, returnEncoded=True).decode()
elif encryptType == 'asym':
if self._utils.validatePubKey(asymPeer):
if stringvalidators.validate_pub_key(asymPeer):
# Encrypt block data with forward secrecy key first, but not meta
jsonMeta = json.dumps(meta)
jsonMeta = self._crypto.pubKeyEncrypt(jsonMeta, asymPeer, encodedData=True).decode()
data = self._crypto.pubKeyEncrypt(data, asymPeer, encodedData=True).decode()
signature = self._crypto.pubKeyEncrypt(signature, asymPeer, encodedData=True).decode()
signer = self._crypto.pubKeyEncrypt(signer, asymPeer, encodedData=True).decode()
try:
onionrusers.OnionrUser(self, asymPeer, saveUser=True)
except ValueError:
# if peer is already known
pass
else:
raise onionrexceptions.InvalidPubkey(asymPeer + ' is not a valid base32 encoded ed25519 key')
@ -786,7 +422,10 @@ class Core:
metadata['expire'] = expire
# send block data (and metadata) to POW module to get tokenized block data
if self.use_subprocess:
payload = subprocesspow.SubprocessPOW(data, metadata, self).start()
else:
payload = onionrproofs.POW(metadata, data).waitForResult()
if payload != False:
try:
retData = self.setData(payload)
@ -794,45 +433,29 @@ class Core:
logger.error(allocationReachedMessage)
retData = False
else:
# Tell the api server through localCommand to wait for the daemon to upload this block to make stastical analysis more difficult
self._utils.localCommand('/waitforshare/' + retData, post=True)
self.addToBlockDB(retData, selfInsert=True, dataSaved=True)
#self.setBlockType(retData, meta['type'])
self._utils.processBlockMetadata(retData)
# Tell the api server through localCommand to wait for the daemon to upload this block to make statistical analysis more difficult
if localcommand.local_command(self, '/ping', maxWait=10) == 'pong!':
if self.config.get('general.security_level', 1) == 0:
localcommand.local_command(self, '/waitforshare/' + retData, post=True, maxWait=5)
self.daemonQueueAdd('uploadBlock', retData)
else:
pass
self.addToBlockDB(retData, selfInsert=True, dataSaved=True)
blockmetadata.process_block_metadata(self, retData)
if retData != False:
if plaintextPeer == 'OVPCZLOXD6DC5JHX4EQ3PSOGAZ3T24F75HQLIUZSDSMYPEOXCPFA====':
events.event('insertdeniable', {'content': plaintext, 'meta': plaintextMeta, 'hash': retData, 'peer': self._utils.bytesToStr(asymPeer)}, onionr = self.onionrInst, threaded = True)
if plaintextPeer == onionrvalues.DENIABLE_PEER_ADDRESS:
events.event('insertdeniable', {'content': plaintext, 'meta': plaintextMeta, 'hash': retData, 'peer': bytesconverter.bytes_to_str(asymPeer)}, onionr = self.onionrInst, threaded = True)
else:
events.event('insertblock', {'content': plaintext, 'meta': plaintextMeta, 'hash': retData, 'peer': self._utils.bytesToStr(asymPeer)}, onionr = self.onionrInst, threaded = True)
events.event('insertblock', {'content': plaintext, 'meta': plaintextMeta, 'hash': retData, 'peer': bytesconverter.bytes_to_str(asymPeer)}, onionr = self.onionrInst, threaded = True)
return retData
def introduceNode(self):
'''
Introduces our node into the network by telling X many nodes our HS address
'''
if(self._utils.isCommunicatorRunning(timeout=30)):
announceAmount = 2
nodeList = self.listAdders()
if len(nodeList) == 0:
for i in self.bootstrapList:
if self._utils.validateID(i):
self.addAddress(i)
nodeList.append(i)
if announceAmount > len(nodeList):
announceAmount = len(nodeList)
for i in range(announceAmount):
self.daemonQueueAdd('announceNode', nodeList[i])
events.event('introduction', onionr = None)
return True
if localcommand.local_command(self, '/ping', maxWait=10) == 'pong!':
self.daemonQueueAdd('announceNode')
logger.info('Introduction command will be processed.', terminal=True)
else:
logger.error('Onionr daemon is not running.')
return False
return
logger.warn('No running node detected. Cannot introduce.', terminal=True)

1
onionr/coredb/__init__.py Normal file
View File

@ -0,0 +1 @@
from . import keydb, blockmetadb, daemonqueue

77
onionr/coredb/blockmetadb/__init__.py Normal file
View File

@ -0,0 +1,77 @@
'''
Onionr - Private P2P Communication
This module works with information relating to blocks stored on the node
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3
from . import expiredblocks, updateblockinfo, add
def get_block_list(core_inst, dateRec = None, unsaved = False):
'''
Get list of our blocks
'''
if dateRec == None:
dateRec = 0
conn = sqlite3.connect(core_inst.blockDB, timeout=30)
c = conn.cursor()
execute = 'SELECT hash FROM hashes WHERE dateReceived >= ? ORDER BY dateReceived ASC;'
args = (dateRec,)
rows = list()
for row in c.execute(execute, args):
for i in row:
rows.append(i)
conn.close()
return rows
def get_block_date(core_inst, blockHash):
'''
Returns the date a block was received
'''
conn = sqlite3.connect(core_inst.blockDB, timeout=30)
c = conn.cursor()
execute = 'SELECT dateReceived FROM hashes WHERE hash=?;'
args = (blockHash,)
for row in c.execute(execute, args):
for i in row:
return int(i)
conn.close()
return None
def get_blocks_by_type(core_inst, blockType, orderDate=True):
'''
Returns a list of blocks by the type
'''
conn = sqlite3.connect(core_inst.blockDB, timeout=30)
c = conn.cursor()
if orderDate:
execute = 'SELECT hash FROM hashes WHERE dataType=? ORDER BY dateReceived;'
else:
execute = 'SELECT hash FROM hashes WHERE dataType=?;'
args = (blockType,)
rows = list()
for row in c.execute(execute, args):
for i in row:
rows.append(i)
conn.close()
return rows

43
onionr/coredb/blockmetadb/add.py Normal file
View File

@ -0,0 +1,43 @@
'''
Onionr - Private P2P Communication
Add an entry to the block metadata database
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import os, sqlite3
from onionrutils import epoch, blockmetadata
def add_to_block_DB(core_inst, newHash, selfInsert=False, dataSaved=False):
'''
Add a hash value to the block db
Should be in hex format!
'''
if not os.path.exists(core_inst.blockDB):
raise Exception('Block db does not exist')
if blockmetadata.has_block(core_inst, newHash):
return
conn = sqlite3.connect(core_inst.blockDB, timeout=30)
c = conn.cursor()
currentTime = epoch.get_epoch() + core_inst._crypto.secrets.randbelow(301)
if selfInsert or dataSaved:
selfInsert = 1
else:
selfInsert = 0
data = (newHash, currentTime, '', selfInsert)
c.execute('INSERT INTO hashes (hash, dateReceived, dataType, dataSaved) VALUES(?, ?, ?, ?);', data)
conn.commit()
conn.close()

35
onionr/coredb/blockmetadb/expiredblocks.py Normal file
View File

@ -0,0 +1,35 @@
'''
Onionr - Private P2P Communication
Get a list of expired blocks still stored
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3
from onionrutils import epoch
def get_expired_blocks(core_inst):
'''Returns a list of expired blocks'''
conn = sqlite3.connect(core_inst.blockDB, timeout=30)
c = conn.cursor()
date = int(epoch.get_epoch())
execute = 'SELECT hash FROM hashes WHERE expire <= %s ORDER BY dateReceived;' % (date,)
rows = list()
for row in c.execute(execute):
for i in row:
rows.append(i)
conn.close()
return rows

33
onionr/coredb/blockmetadb/updateblockinfo.py Normal file
View File

@ -0,0 +1,33 @@
'''
Onionr - Private P2P Communication
Update block information in the metadata database by a field name
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3
def update_block_info(core_inst, hash, key, data):
if key not in ('dateReceived', 'decrypted', 'dataType', 'dataFound', 'dataSaved', 'sig', 'author', 'dateClaimed', 'expire'):
return False
conn = sqlite3.connect(core_inst.blockDB, timeout=30)
c = conn.cursor()
args = (data, hash)
c.execute("UPDATE hashes SET " + key + " = ? where hash = ?;", args)
conn.commit()
conn.close()
return True

97
onionr/coredb/daemonqueue/__init__.py Normal file
View File

@ -0,0 +1,97 @@
'''
Onionr - Private P2P Communication
Write and read the daemon queue, which is how messages are passed into the onionr daemon in a more
direct way than the http api
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3, os
import onionrevents as events
from onionrutils import localcommand, epoch
def daemon_queue(core_inst):
'''
Gives commands to the communication proccess/daemon by reading an sqlite3 database
This function intended to be used by the client. Queue to exchange data between "client" and server.
'''
retData = False
if not os.path.exists(core_inst.queueDB):
core_inst.dbCreate.createDaemonDB()
else:
conn = sqlite3.connect(core_inst.queueDB, timeout=30)
c = conn.cursor()
try:
for row in c.execute('SELECT command, data, date, min(ID), responseID FROM commands group by id'):
retData = row
break
except sqlite3.OperationalError:
core_inst.dbCreate.createDaemonDB()
else:
if retData != False:
c.execute('DELETE FROM commands WHERE id=?;', (retData[3],))
conn.commit()
conn.close()
events.event('queue_pop', data = {'data': retData}, onionr = core_inst.onionrInst)
return retData
def daemon_queue_add(core_inst, command, data='', responseID=''):
'''
Add a command to the daemon queue, used by the communication daemon (communicator.py)
'''
retData = True
date = epoch.get_epoch()
conn = sqlite3.connect(core_inst.queueDB, timeout=30)
c = conn.cursor()
t = (command, data, date, responseID)
try:
c.execute('INSERT INTO commands (command, data, date, responseID) VALUES(?, ?, ?, ?)', t)
conn.commit()
except sqlite3.OperationalError:
retData = False
core_inst.daemonQueue()
events.event('queue_push', data = {'command': command, 'data': data}, onionr = core_inst.onionrInst)
conn.close()
return retData
def daemon_queue_get_response(core_inst, responseID=''):
'''
Get a response sent by communicator to the API, by requesting to the API
'''
assert len(responseID) > 0
resp = localcommand.local_command(core_inst, 'queueResponse/' + responseID)
return resp
def clear_daemon_queue(core_inst):
'''
Clear the daemon queue (somewhat dangerous)
'''
conn = sqlite3.connect(core_inst.queueDB, timeout=30)
c = conn.cursor()
try:
c.execute('DELETE FROM commands;')
conn.commit()
except:
pass
conn.close()
events.event('queue_clear', onionr = core_inst.onionrInst)

1
onionr/coredb/keydb/__init__.py Normal file
View File

@ -0,0 +1 @@
from . import addkeys, listkeys, removekeys, userinfo, transportinfo

91
onionr/coredb/keydb/addkeys.py Normal file
View File

@ -0,0 +1,91 @@
'''
Onionr - Private P2P Communication
add user keys or transport addresses
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3
import onionrevents as events
from onionrutils import stringvalidators
def add_peer(core_inst, peerID, name=''):
'''
Adds a public key to the key database (misleading function name)
'''
if peerID in core_inst.listPeers() or peerID == core_inst._crypto.pubKey:
raise ValueError("specified id is already known")
# This function simply adds a peer to the DB
if not stringvalidators.validate_pub_key(peerID):
return False
events.event('pubkey_add', data = {'key': peerID}, onionr = core_inst.onionrInst)
conn = sqlite3.connect(core_inst.peerDB, timeout=30)
hashID = core_inst._crypto.pubKeyHashID(peerID)
c = conn.cursor()
t = (peerID, name, 'unknown', hashID, 0)
for i in c.execute("SELECT * FROM peers WHERE id = ?;", (peerID,)):
try:
if i[0] == peerID:
conn.close()
return False
except ValueError:
pass
except IndexError:
pass
c.execute('INSERT INTO peers (id, name, dateSeen, hashID, trust) VALUES(?, ?, ?, ?, ?);', t)
conn.commit()
conn.close()
return True
def add_address(core_inst, address):
'''
Add an address to the address database (only tor currently)
'''
if type(address) is None or len(address) == 0:
return False
if stringvalidators.validate_transport(address):
if address == core_inst.config.get('i2p.ownAddr', None) or address == core_inst.hsAddress:
return False
conn = sqlite3.connect(core_inst.addressDB, timeout=30)
c = conn.cursor()
# check if address is in database
# this is safe to do because the address is validated above, but we strip some chars here too just in case
address = address.replace('\'', '').replace(';', '').replace('"', '').replace('\\', '')
for i in c.execute("SELECT * FROM adders WHERE address = ?;", (address,)):
try:
if i[0] == address:
conn.close()
return False
except ValueError:
pass
except IndexError:
pass
t = (address, 1)
c.execute('INSERT INTO adders (address, type) VALUES(?, ?);', t)
conn.commit()
conn.close()
events.event('address_add', data = {'address': address}, onionr = core_inst.onionrInst)
return True
else:
return False

83
onionr/coredb/keydb/listkeys.py Normal file
View File

@ -0,0 +1,83 @@
'''
Onionr - Private P2P Communication
get lists for user keys or transport addresses
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3
import logger
from onionrutils import epoch
def list_peers(core_inst, randomOrder=True, getPow=False, trust=0):
'''
Return a list of public keys (misleading function name)
randomOrder determines if the list should be in a random order
trust sets the minimum trust to list
'''
conn = sqlite3.connect(core_inst.peerDB, timeout=30)
c = conn.cursor()
payload = ''
if trust not in (0, 1, 2):
logger.error('Tried to select invalid trust.')
return
if randomOrder:
payload = 'SELECT * FROM peers WHERE trust >= ? ORDER BY RANDOM();'
else:
payload = 'SELECT * FROM peers WHERE trust >= ?;'
peerList = []
for i in c.execute(payload, (trust,)):
try:
if len(i[0]) != 0:
if getPow:
peerList.append(i[0] + '-' + i[1])
else:
peerList.append(i[0])
except TypeError:
pass
conn.close()
return peerList
def list_adders(core_inst, randomOrder=True, i2p=True, recent=0):
'''
Return a list of transport addresses
'''
conn = sqlite3.connect(core_inst.addressDB, timeout=30)
c = conn.cursor()
if randomOrder:
addresses = c.execute('SELECT * FROM adders ORDER BY RANDOM();')
else:
addresses = c.execute('SELECT * FROM adders;')
addressList = []
for i in addresses:
if len(i[0].strip()) == 0:
continue
addressList.append(i[0])
conn.close()
testList = list(addressList) # create new list to iterate
for address in testList:
try:
if recent > 0 and (epoch.get_epoch() - core_inst.getAddressInfo(address, 'lastConnect')) > recent:
raise TypeError # If there is no last-connected date or it was too long ago, don't add peer to list if recent is not 0
except TypeError:
addressList.remove(address)
return addressList

40
onionr/coredb/keydb/removekeys.py Normal file
View File

@ -0,0 +1,40 @@
'''
Onionr - Private P2P Communication
Remove a transport address but don't ban them
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3
import onionrevents as events
from onionrutils import stringvalidators
def remove_address(core_inst, address):
'''
Remove an address from the address database
'''
if stringvalidators.validate_transport(address):
conn = sqlite3.connect(core_inst.addressDB, timeout=30)
c = conn.cursor()
t = (address,)
c.execute('Delete from adders where address=?;', t)
conn.commit()
conn.close()
events.event('address_remove', data = {'address': address}, onionr = core_inst.onionrInst)
return True
else:
return False

72
onionr/coredb/keydb/transportinfo.py Normal file
View File

@ -0,0 +1,72 @@
'''
Onionr - Private P2P Communication
get or set transport address meta information
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3
def get_address_info(core_inst, address, info):
'''
Get info about an address from its database entry
address text, 0
type int, 1
knownPeer text, 2
speed int, 3
success int, 4
powValue 5
failure int 6
lastConnect 7
trust 8
introduced 9
'''
conn = sqlite3.connect(core_inst.addressDB, timeout=30)
c = conn.cursor()
command = (address,)
infoNumbers = {'address': 0, 'type': 1, 'knownPeer': 2, 'speed': 3, 'success': 4, 'powValue': 5, 'failure': 6, 'lastConnect': 7, 'trust': 8, 'introduced': 9}
info = infoNumbers[info]
iterCount = 0
retVal = ''
for row in c.execute('SELECT * FROM adders WHERE address=?;', command):
for i in row:
if iterCount == info:
retVal = i
break
else:
iterCount += 1
conn.close()
return retVal
def set_address_info(core_inst, address, key, data):
'''
Update an address for a key
'''
conn = sqlite3.connect(core_inst.addressDB, timeout=30)
c = conn.cursor()
command = (data, address)
if key not in ('address', 'type', 'knownPeer', 'speed', 'success', 'failure', 'powValue', 'lastConnect', 'lastConnectAttempt', 'trust', 'introduced'):
raise Exception("Got invalid database key when setting address info")
else:
c.execute('UPDATE adders SET ' + key + ' = ? WHERE address=?', command)
conn.commit()
conn.close()

69
onionr/coredb/keydb/userinfo.py Normal file
View File

@ -0,0 +1,69 @@
'''
Onionr - Private P2P Communication
get or set information about a user id
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sqlite3
def get_user_info(core_inst, peer, info):
'''
Get info about a peer from their database entry
id text 0
name text, 1
adders text, 2
dateSeen not null, 3
trust int 4
hashID text 5
'''
conn = sqlite3.connect(core_inst.peerDB, timeout=30)
c = conn.cursor()
command = (peer,)
infoNumbers = {'id': 0, 'name': 1, 'adders': 2, 'dateSeen': 3, 'trust': 4, 'hashID': 5}
info = infoNumbers[info]
iterCount = 0
retVal = ''
for row in c.execute('SELECT * FROM peers WHERE id=?;', command):
for i in row:
if iterCount == info:
retVal = i
break
else:
iterCount += 1
conn.close()
return retVal
def set_peer_info(core_inst, peer, key, data):
'''
Update a peer for a key
'''
conn = sqlite3.connect(core_inst.peerDB, timeout=30)
c = conn.cursor()
command = (data, peer)
# TODO: validate key on whitelist
if key not in ('id', 'name', 'pubkey', 'forwardKey', 'dateSeen', 'trust'):
raise Exception("Got invalid database key when setting peer info")
c.execute('UPDATE peers SET ' + key + ' = ? WHERE id=?', command)
conn.commit()
conn.close()

2
onionr/dbcreator.py
View File

@ -1,5 +1,5 @@
'''
Onionr - P2P Anonymous Data Storage & Sharing
Onionr - Private P2P Communication
DBCreator, creates sqlite3 databases used by Onionr
'''

331
onionr/dependencies/secrets.py
View File

@ -1,331 +0,0 @@
"""Generate cryptographically strong pseudo-random numbers suitable for
managing secrets such as account authentication, tokens, and similar.
See PEP 506 for more information.
https://www.python.org/dev/peps/pep-0506/
A. HISTORY OF THE SOFTWARE
==========================
Python was created in the early 1990s by Guido van Rossum at Stichting
Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands
as a successor of a language called ABC. Guido remains Python's
principal author, although it includes many contributions from others.
In 1995, Guido continued his work on Python at the Corporation for
National Research Initiatives (CNRI, see http://www.cnri.reston.va.us)
in Reston, Virginia where he released several versions of the
software.
In May 2000, Guido and the Python core development team moved to
BeOpen.com to form the BeOpen PythonLabs team. In October of the same
year, the PythonLabs team moved to Digital Creations, which became
Zope Corporation. In 2001, the Python Software Foundation (PSF, see
https://www.python.org/psf/) was formed, a non-profit organization
created specifically to own Python-related Intellectual Property.
Zope Corporation was a sponsoring member of the PSF.
All Python releases are Open Source (see http://www.opensource.org for
the Open Source Definition). Historically, most, but not all, Python
releases have also been GPL-compatible; the table below summarizes
the various releases.
Release Derived Year Owner GPL-
from compatible? (1)
0.9.0 thru 1.2 1991-1995 CWI yes
1.3 thru 1.5.2 1.2 1995-1999 CNRI yes
1.6 1.5.2 2000 CNRI no
2.0 1.6 2000 BeOpen.com no
1.6.1 1.6 2001 CNRI yes (2)
2.1 2.0+1.6.1 2001 PSF no
2.0.1 2.0+1.6.1 2001 PSF yes
2.1.1 2.1+2.0.1 2001 PSF yes
2.1.2 2.1.1 2002 PSF yes
2.1.3 2.1.2 2002 PSF yes
2.2 and above 2.1.1 2001-now PSF yes
Footnotes:
(1) GPL-compatible doesn't mean that we're distributing Python under
the GPL. All Python licenses, unlike the GPL, let you distribute
a modified version without making your changes open source. The
GPL-compatible licenses make it possible to combine Python with
other software that is released under the GPL; the others don't.
(2) According to Richard Stallman, 1.6.1 is not GPL-compatible,
because its license has a choice of law clause. According to
CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1
is "not incompatible" with the GPL.
Thanks to the many outside volunteers who have worked under Guido's
direction to make these releases possible.
B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON
===============================================================
PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
--------------------------------------------
1. This LICENSE AGREEMENT is between the Python Software Foundation
("PSF"), and the Individual or Organization ("Licensee") accessing and
otherwise using this software ("Python") in source or binary form and
its associated documentation.
2. Subject to the terms and conditions of this License Agreement, PSF hereby
grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce,
analyze, test, perform and/or display publicly, prepare derivative works,
distribute, and otherwise use Python alone or in any derivative version,
provided, however, that PSF's License Agreement and PSF's notice of copyright,
i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010,
2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation; All
Rights Reserved" are retained in Python alone or in any derivative version
prepared by Licensee.
3. In the event Licensee prepares a derivative work that is based on
or incorporates Python or any part thereof, and wants to make
the derivative work available to others as provided herein, then
Licensee hereby agrees to include in any such work a brief summary of
the changes made to Python.
4. PSF is making Python available to Licensee on an "AS IS"
basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT
INFRINGE ANY THIRD PARTY RIGHTS.
5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON,
OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
6. This License Agreement will automatically terminate upon a material
breach of its terms and conditions.
7. Nothing in this License Agreement shall be deemed to create any
relationship of agency, partnership, or joint venture between PSF and
Licensee. This License Agreement does not grant permission to use PSF
trademarks or trade name in a trademark sense to endorse or promote
products or services of Licensee, or any third party.
8. By copying, installing or otherwise using Python, Licensee
agrees to be bound by the terms and conditions of this License
Agreement.
BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0
-------------------------------------------
BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1
1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an
office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the
Individual or Organization ("Licensee") accessing and otherwise using
this software in source or binary form and its associated
documentation ("the Software").
2. Subject to the terms and conditions of this BeOpen Python License
Agreement, BeOpen hereby grants Licensee a non-exclusive,
royalty-free, world-wide license to reproduce, analyze, test, perform
and/or display publicly, prepare derivative works, distribute, and
otherwise use the Software alone or in any derivative version,
provided, however, that the BeOpen Python License is retained in the
Software, alone or in any derivative version prepared by Licensee.
3. BeOpen is making the Software available to Licensee on an "AS IS"
basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT
INFRINGE ANY THIRD PARTY RIGHTS.
4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE
SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS
AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY
DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
5. This License Agreement will automatically terminate upon a material
breach of its terms and conditions.
6. This License Agreement shall be governed by and interpreted in all
respects by the law of the State of California, excluding conflict of
law provisions. Nothing in this License Agreement shall be deemed to
create any relationship of agency, partnership, or joint venture
between BeOpen and Licensee. This License Agreement does not grant
permission to use BeOpen trademarks or trade names in a trademark
sense to endorse or promote products or services of Licensee, or any
third party. As an exception, the "BeOpen Python" logos available at
http://www.pythonlabs.com/logos.html may be used according to the
permissions granted on that web page.
7. By copying, installing or otherwise using the software, Licensee
agrees to be bound by the terms and conditions of this License
Agreement.
CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1
---------------------------------------
1. This LICENSE AGREEMENT is between the Corporation for National
Research Initiatives, having an office at 1895 Preston White Drive,
Reston, VA 20191 ("CNRI"), and the Individual or Organization
("Licensee") accessing and otherwise using Python 1.6.1 software in
source or binary form and its associated documentation.
2. Subject to the terms and conditions of this License Agreement, CNRI
hereby grants Licensee a nonexclusive, royalty-free, world-wide
license to reproduce, analyze, test, perform and/or display publicly,
prepare derivative works, distribute, and otherwise use Python 1.6.1
alone or in any derivative version, provided, however, that CNRI's
License Agreement and CNRI's notice of copyright, i.e., "Copyright (c)
1995-2001 Corporation for National Research Initiatives; All Rights
Reserved" are retained in Python 1.6.1 alone or in any derivative
version prepared by Licensee. Alternately, in lieu of CNRI's License
Agreement, Licensee may substitute the following text (omitting the
quotes): "Python 1.6.1 is made available subject to the terms and
conditions in CNRI's License Agreement. This Agreement together with
Python 1.6.1 may be located on the Internet using the following
unique, persistent identifier (known as a handle): 1895.22/1013. This
Agreement may also be obtained from a proxy server on the Internet
using the following URL: http://hdl.handle.net/1895.22/1013".
3. In the event Licensee prepares a derivative work that is based on
or incorporates Python 1.6.1 or any part thereof, and wants to make
the derivative work available to others as provided herein, then
Licensee hereby agrees to include in any such work a brief summary of
the changes made to Python 1.6.1.
4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS"
basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT
INFRINGE ANY THIRD PARTY RIGHTS.
5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1,
OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
6. This License Agreement will automatically terminate upon a material
breach of its terms and conditions.
7. This License Agreement shall be governed by the federal
intellectual property law of the United States, including without
limitation the federal copyright law, and, to the extent such
U.S. federal law does not apply, by the law of the Commonwealth of
Virginia, excluding Virginia's conflict of law provisions.
Notwithstanding the foregoing, with regard to derivative works based
on Python 1.6.1 that incorporate non-separable material that was
previously distributed under the GNU General Public License (GPL), the
law of the Commonwealth of Virginia shall govern this License
Agreement only as to issues arising under or with respect to
Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this
License Agreement shall be deemed to create any relationship of
agency, partnership, or joint venture between CNRI and Licensee. This
License Agreement does not grant permission to use CNRI trademarks or
trade name in a trademark sense to endorse or promote products or
services of Licensee, or any third party.
8. By clicking on the "ACCEPT" button where indicated, or by copying,
installing or otherwise using Python 1.6.1, Licensee agrees to be
bound by the terms and conditions of this License Agreement.
ACCEPT
CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2
--------------------------------------------------
Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam,
The Netherlands. All rights reserved.
Permission to use, copy, modify, and distribute this software and its
documentation for any purpose and without fee is hereby granted,
provided that the above copyright notice appear in all copies and that
both that copyright notice and this permission notice appear in
supporting documentation, and that the name of Stichting Mathematisch
Centrum or CWI not be used in advertising or publicity pertaining to
distribution of the software without specific, written prior
permission.
STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO
THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE
FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
"""
__all__ = ['choice', 'randbelow', 'randbits', 'SystemRandom',
'token_bytes', 'token_hex', 'token_urlsafe',
'compare_digest',
]
import base64
import binascii
import os
from hmac import compare_digest
from random import SystemRandom
_sysrand = SystemRandom()
randbits = _sysrand.getrandbits
choice = _sysrand.choice
def randbelow(exclusive_upper_bound):
"""Return a random int in the range [0, n)."""
if exclusive_upper_bound <= 0:
raise ValueError("Upper bound must be positive.")
return _sysrand._randbelow(exclusive_upper_bound)
DEFAULT_ENTROPY = 32 # number of bytes to return by default
def token_bytes(nbytes=None):
"""Return a random byte string containing *nbytes* bytes.
If *nbytes* is ``None`` or not supplied, a reasonable
default is used.
>>> token_bytes(16) #doctest:+SKIP
b'\\xebr\\x17D*t\\xae\\xd4\\xe3S\\xb6\\xe2\\xebP1\\x8b'
"""
if nbytes is None:
nbytes = DEFAULT_ENTROPY
return os.urandom(nbytes)
def token_hex(nbytes=None):
"""Return a random text string, in hexadecimal.
The string has *nbytes* random bytes, each byte converted to two
hex digits. If *nbytes* is ``None`` or not supplied, a reasonable
default is used.
>>> token_hex(16) #doctest:+SKIP
'f9bf78b9a18ce6d46a0cd2b0b86df9da'
"""
return binascii.hexlify(token_bytes(nbytes)).decode('ascii')
def token_urlsafe(nbytes=None):
"""Return a random URL-safe text string, in Base64 encoding.
The string has *nbytes* random bytes. If *nbytes* is ``None``
or not supplied, a reasonable default is used.
>>> token_urlsafe(16) #doctest:+SKIP
'Drmhze6EPcv0fN_81Bj-nA'
"""
tok = token_bytes(nbytes)
return base64.urlsafe_b64encode(tok).rstrip(b'=').decode('ascii')

11
onionr/etc/README.md Executable file
View File

@ -0,0 +1,11 @@
# etc
Files that don't really fit anywhere else, but aren't used very frequently.
## Files
humanreadabletime.py: take integer seconds and return a human readable time string
pgpwords.py: represent data using the pgp word list: https://en.wikipedia.org/wiki/PGP_word_list
onionrvalues.py: spec values for onionr blocks and other things

38
onionr/etc/humanreadabletime.py Executable file
View File

@ -0,0 +1,38 @@
'''
Onionr - Private P2P Communication
human_readable_time takes integer seconds and returns a human readable string
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
def human_readable_time(seconds):
build = ''
units = {
'year' : 31557600,
'month' : (31557600 / 12),
'day' : 86400,
'hour' : 3600,
'minute' : 60,
'second' : 1
}
for unit in units:
amnt_unit = int(seconds / units[unit])
if amnt_unit >= 1:
seconds -= amnt_unit * units[unit]
build += '%s %s' % (amnt_unit, unit) + ('s' if amnt_unit != 1 else '') + ' '
return build.strip()

8
onionr/etc/onionrvalues.py
View File

@ -1,5 +1,5 @@
'''
Onionr - P2P Microblogging Platform & Social network
Onionr - Private P2P Communication
This file defines values and requirements used by Onionr
'''
@ -17,8 +17,10 @@
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
DENIABLE_PEER_ADDRESS = "OVPCZLOXD6DC5JHX4EQ3PSOGAZ3T24F75HQLIUZSDSMYPEOXCPFA===="
class OnionrValues:
def __init__(self):
self.passwordLength = 20
self.blockMetadataLengths = {'meta': 1000, 'sig': 200, 'signer': 200, 'time': 10, 'powRandomToken': 1000, 'encryptType': 4, 'expire': 14} #TODO properly refine values to minimum needed
self.blockMetadataLengths = {'meta': 1000, 'sig': 200, 'signer': 200, 'time': 10, 'pow': 1000, 'encryptType': 4, 'expire': 14} #TODO properly refine values to minimum needed
self.default_expire = 2592000
self.announce_pow = 5

8
onionr/etc/pgpwords.py
View File

@ -3,6 +3,14 @@
'''This file is adapted from https://github.com/thblt/pgp-words by github user 'thblt' ('Thibault Polge), GPL v3 license'''
'''
Changes made for Onionr by Kevin Froman in 2018-2019:
Minor changes such as slight word adjustment, line breaks
CLI commands/usage function removed
hexify function added
'''
import os, re, sys, binascii
_words = [

27
onionr/etc/powchoice.py Executable file
View File

@ -0,0 +1,27 @@
'''
Onionr - Private P2P Communication
This file does determinations for what proof of work module should be used
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import platform
def use_subprocess(core_inst):
use = True
if not core_inst.config.get('general.use_subprocess_pow_if_possible', True):
use = False
if 'Windows' == platform.system():
use = False
return use

13
onionr/httpapi/README.md Executable file
View File

@ -0,0 +1,13 @@
# httpapi
The httpapi contains collections of endpoints for the client and public API servers.
## Files:
configapi: manage onionr configuration from the client http api
friendsapi: add, remove and list friends from the client http api
miscpublicapi: misculanious onionr network interaction from the **public** httpapi, such as announcements, block fetching and uploading.
profilesapi: work in progress in returning a profile page for an Onionr user

6
onionr/httpapi/__init__.py
View File

@ -1,5 +1,5 @@
'''
Onionr - P2P Anonymous Storage Network
Onionr - Private P2P Communication
This file registers plugin's flask blueprints for the client http server
'''
@ -19,11 +19,11 @@
'''
import onionrplugins
def load_plugin_blueprints(flaskapp):
def load_plugin_blueprints(flaskapp, blueprint='flask_blueprint'):
'''Iterate enabled plugins and load any http endpoints they have'''
for plugin in onionrplugins.get_enabled_plugins():
plugin = onionrplugins.get_plugin(plugin)
try:
flaskapp.register_blueprint(getattr(plugin, 'flask_blueprint'))
flaskapp.register_blueprint(getattr(plugin, blueprint))
except AttributeError:
pass

3
onionr/httpapi/apiutils/__init__.py Normal file
View File

@ -0,0 +1,3 @@
from . import shutdown, setbindip, getblockdata
GetBlockData = getblockdata.GetBlockData

42
onionr/httpapi/apiutils/getblockdata.py Normal file
View File

@ -0,0 +1,42 @@
import json
import core, onionrblockapi
from onionrutils import bytesconverter, stringvalidators
class GetBlockData:
def __init__(self, client_api_inst=None):
if client_api_inst is None:
self.client_api_inst = None
self.c = core.Core()
elif isinstance(client_api_inst, core.Core):
self.client_api_inst = None
self.c = client_api_inst
else:
self.client_api_Inst = client_api_inst
self.c = core.Core()
def get_block_data(self, bHash, decrypt=False, raw=False, headerOnly=False):
assert stringvalidators.validate_hash(bHash)
bl = onionrblockapi.Block(bHash, core=self.c)
if decrypt:
bl.decrypt()
if bl.isEncrypted and not bl.decrypted:
raise ValueError
if not raw:
if not headerOnly:
retData = {'meta':bl.bheader, 'metadata': bl.bmetadata, 'content': bl.bcontent}
for x in list(retData.keys()):
try:
retData[x] = retData[x].decode()
except AttributeError:
pass
else:
validSig = False
signer = bytesconverter.bytes_to_str(bl.signer)
if bl.isSigned() and stringvalidators.validate_pub_key(signer) and bl.isSigner(signer):
validSig = True
bl.bheader['validSig'] = validSig
bl.bheader['meta'] = ''
retData = {'meta': bl.bheader, 'metadata': bl.bmetadata}
return json.dumps(retData)
else:
return bl.raw

25
onionr/httpapi/apiutils/setbindip.py Normal file
View File

@ -0,0 +1,25 @@
import random, socket
import config, logger
def set_bind_IP(filePath='', core_inst=None):
'''Set a random localhost IP to a specified file (intended for private or public API localhost IPs)'''
if not core_inst is None:
config = core_inst.config
if config.get('general.random_bind_ip', True):
hostOctets = [str(127), str(random.randint(0x02, 0xFF)), str(random.randint(0x02, 0xFF)), str(random.randint(0x02, 0xFF))]
data = '.'.join(hostOctets)
# Try to bind IP. Some platforms like Mac block non normal 127.x.x.x
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
s.bind((data, 0))
except OSError:
# if mac/non-bindable, show warning and default to 127.0.0.1
logger.warn('Your platform appears to not support random local host addresses 127.x.x.x. Falling back to 127.0.0.1.')
data = '127.0.0.1'
s.close()
else:
data = '127.0.0.1'
if filePath != '':
with open(filePath, 'w') as bindFile:
bindFile.write(data)
return data

38
onionr/httpapi/apiutils/shutdown.py Normal file
View File

@ -0,0 +1,38 @@
'''
Onionr - Private P2P Communication
Shutdown the node either hard or cleanly
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
from flask import Blueprint, Response
import core, onionrblockapi, onionrexceptions
from onionrutils import stringvalidators
shutdown_bp = Blueprint('shutdown', __name__)
def shutdown(client_api_inst):
try:
client_api_inst.publicAPI.httpServer.stop()
client_api_inst.httpServer.stop()
except AttributeError:
pass
return Response("bye")
@shutdown_bp.route('/shutdownclean')
def shutdown_clean():
# good for calling from other clients
core.Core().daemonQueueAdd('shutdown')
return Response("bye")

62
onionr/httpapi/configapi/__init__.py Executable file
View File

@ -0,0 +1,62 @@
'''
Onionr - Private P2P Communication
This file handles configuration setting and getting from the HTTP API
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import json
from flask import Blueprint, request, Response, abort
import config, onionrutils
config.reload()
config_BP = Blueprint('config_BP', __name__)
@config_BP.route('/config/get')
def get_all_config():
'''Simply return all configuration as JSON string'''
return Response(json.dumps(config.get_config(), indent=4, sort_keys=True))
@config_BP.route('/config/get/<key>')
def get_by_key(key):
'''Return a config setting by key'''
return Response(json.dumps(config.get(key)))
@config_BP.route('/config/setall', methods=['POST'])
def set_all_config():
'''Overwrite existing JSON config with new JSON string'''
try:
new_config = request.get_json(force=True)
except json.JSONDecodeError:
abort(400)
else:
config.set_config(new_config)
config.save()
return Response('success')
@config_BP.route('/config/set/<key>', methods=['POST'])
def set_by_key(key):
'''Overwrite/set only 1 config key'''
'''
{
'data': data
}
'''
try:
data = json.loads(onionrutils.OnionrUtils.bytesToStr(request.data))['data']
except (json.JSONDecodeError, KeyError):
abort(400)
config.set(key, data, True)
return Response('success')

21
onionr/httpapi/fdsafehandler.py Normal file
View File

@ -0,0 +1,21 @@
from gevent.pywsgi import WSGIServer, WSGIHandler
from gevent import Timeout
class FDSafeHandler(WSGIHandler):
'''Our WSGI handler. Doesn't do much non-default except timeouts'''
def handle(self):
self.timeout = Timeout(120, Exception)
self.timeout.start()
try:
WSGIHandler.handle(self)
except Exception:
self.handle_error()
finally:
self.timeout.close()
def handle_error(self):
if v is self.timeout:
self.result = [b"Timeout"]
self.start_response("200 OK", [])
self.process_result()
else:
WSGIHandler.handle_error(self)

5
onionr/httpapi/friendsapi/__init__.py
View File

@ -1,5 +1,5 @@
'''
Onionr - P2P Anonymous Storage Network
Onionr - Private P2P Communication
This file creates http endpoints for friend management
'''
@ -26,7 +26,8 @@ friends = Blueprint('friends', __name__)
@friends.route('/friends/list')
def list_friends():
pubkey_list = {}
friend_list = contactmanager.ContactManager.list_friends(core.Core())
c = core.Core()
friend_list = contactmanager.ContactManager.list_friends(c)
for friend in friend_list:
pubkey_list[friend.publicKey] = {'name': friend.get_info('name')}
return json.dumps(pubkey_list)

63
onionr/httpapi/insertblock.py Normal file
View File

@ -0,0 +1,63 @@
'''
Onionr - Private P2P Communication
Create blocks with the client api server
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import json, threading
from flask import Blueprint, Response, request
import core
ib = Blueprint('insertblock', __name__)
@ib.route('/insertblock', methods=['POST'])
def client_api_insert_block():
c = core.Core()
encrypt = False
bData = request.get_json(force=True)
message = bData['message']
# Detect if message (block body) is not specified
if type(message) is None:
return 'failure', 406
subject = 'temp'
encryptType = ''
sign = True
meta = {}
to = ''
try:
if bData['encrypt']:
to = bData['to']
encrypt = True
encryptType = 'asym'
except KeyError:
pass
try:
if not bData['sign']:
sign = False
except KeyError:
pass
try:
bType = bData['type']
except KeyError:
bType = 'bin'
try:
meta = json.loads(bData['meta'])
except KeyError:
pass
threading.Thread(target=c.insertBlock, args=(message,), kwargs={'header': bType, 'encryptType': encryptType, 'sign':sign, 'asymPeer': to, 'meta': meta}).start()
return Response('success')

1
onionr/httpapi/miscclientapi/__init__.py Normal file
View File

@ -0,0 +1 @@
from . import getblocks, staticfiles, endpoints

112
onionr/httpapi/miscclientapi/endpoints.py Normal file
View File

@ -0,0 +1,112 @@
'''
Onionr - Private P2P Communication
Misc client API endpoints too small to need their own file and that need access to the client api inst
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
from flask import Response, Blueprint, request, send_from_directory, abort
from httpapi import apiutils
class PrivateEndpoints:
def __init__(self, client_api):
private_endpoints_bp = Blueprint('privateendpoints', __name__)
self.private_endpoints_bp = private_endpoints_bp
config = client_api._core.config
@private_endpoints_bp.route('/serviceactive/<pubkey>')
def serviceActive(pubkey):
try:
if pubkey in client_api._core.onionrInst.communicatorInst.active_services:
return Response('true')
except AttributeError as e:
pass
return Response('false')
@private_endpoints_bp.route('/www/<path:path>', endpoint='www')
def wwwPublic(path):
if not config.get("www.private.run", True):
abort(403)
return send_from_directory(config.get('www.private.path', 'static-data/www/private/'), path)
@private_endpoints_bp.route('/hitcount')
def get_hit_count():
return Response(str(client_api.publicAPI.hitCount))
@private_endpoints_bp.route('/queueResponseAdd/<name>', methods=['post'])
def queueResponseAdd(name):
# Responses from the daemon. TODO: change to direct var access instead of http endpoint
client_api.queueResponse[name] = request.form['data']
return Response('success')
@private_endpoints_bp.route('/queueResponse/<name>')
def queueResponse(name):
# Fetch a daemon queue response
resp = 'failure'
try:
resp = client_api.queueResponse[name]
except KeyError:
pass
else:
del client_api.queueResponse[name]
if resp == 'failure':
return resp, 404
else:
return resp
@private_endpoints_bp.route('/ping')
def ping():
# Used to check if client api is working
return Response("pong!")
@private_endpoints_bp.route('/lastconnect')
def lastConnect():
return Response(str(client_api.publicAPI.lastRequest))
@private_endpoints_bp.route('/waitforshare/<name>', methods=['post'])
def waitforshare(name):
'''Used to prevent the **public** api from sharing blocks we just created'''
assert name.isalnum()
if name in client_api.publicAPI.hideBlocks:
client_api.publicAPI.hideBlocks.remove(name)
return Response("removed")
else:
client_api.publicAPI.hideBlocks.append(name)
return Response("added")
@private_endpoints_bp.route('/shutdown')
def shutdown():
return apiutils.shutdown.shutdown(client_api)
@private_endpoints_bp.route('/getstats')
def getStats():
# returns node stats
#return Response("disabled")
while True:
try:
return Response(client_api._core.serializer.getStats())
except AttributeError:
pass
@private_endpoints_bp.route('/getuptime')
def showUptime():
return Response(str(client_api.getUptime()))
@private_endpoints_bp.route('/getActivePubkey')
def getActivePubkey():
return Response(client_api._core._crypto.pubKey)
@private_endpoints_bp.route('/getHumanReadable/<name>')
def getHumanReadable(name):
return Response(mnemonickeys.get_human_readable_ID(name))

66
onionr/httpapi/miscclientapi/getblocks.py Normal file
View File

@ -0,0 +1,66 @@
'''
Onionr - Private P2P Communication
Create blocks with the client api server
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
from flask import Blueprint, Response, abort
import core, onionrblockapi
from httpapi import apiutils
from onionrutils import stringvalidators
c = core.Core()
client_get_block = apiutils.GetBlockData(c)
client_get_blocks = Blueprint('miscclient', __name__)
@client_get_blocks.route('/getblocksbytype/<name>')
def getBlocksByType(name):
blocks = c.getBlocksByType(name)
return Response(','.join(blocks))
@client_get_blocks.route('/getblockbody/<name>')
def getBlockBodyData(name):
resp = ''
if stringvalidators.validate_hash(name):
try:
resp = onionrblockapi.Block(name, decrypt=True, core=c).bcontent
except TypeError:
pass
else:
abort(404)
return Response(resp)
@client_get_blocks.route('/getblockdata/<name>')
def getData(name):
resp = ""
if stringvalidators.validate_hash(name):
if name in c.getBlockList():
try:
resp = client_get_block.get_block_data(name, decrypt=True)
except ValueError:
pass
else:
abort(404)
else:
abort(404)
return Response(resp)
@client_get_blocks.route('/getblockheader/<name>')
def getBlockHeader(name):
resp = client_get_block.get_block_data(name, decrypt=True, headerOnly=True)
return Response(resp)

70
onionr/httpapi/miscclientapi/staticfiles.py Normal file
View File

@ -0,0 +1,70 @@
'''
Onionr - Private P2P Communication
Register static file routes
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import os
from flask import Blueprint, send_from_directory
static_files_bp = Blueprint('staticfiles', __name__)
root = os.getcwd() + '/static-data/www/' # should be set to onionr install directory from onionr startup
@static_files_bp.route('/board/', endpoint='board')
def loadBoard():
return send_from_directory(root + 'board/', "index.html")
@static_files_bp.route('/mail/<path:path>', endpoint='mail')
def loadMail(path):
return send_from_directory(root + 'mail/', path)
@static_files_bp.route('/mail/', endpoint='mailindex')
def loadMailIndex():
return send_from_directory(root + 'mail/', 'index.html')
@static_files_bp.route('/friends/<path:path>', endpoint='friends')
def loadContacts(path):
return send_from_directory(root + 'friends/', path)
@static_files_bp.route('/friends/', endpoint='friendsindex')
def loadContacts():
return send_from_directory(root + 'friends/', 'index.html')
@static_files_bp.route('/profiles/<path:path>', endpoint='profiles')
def loadContacts(path):
return send_from_directory(root + 'profiles/', path)
@static_files_bp.route('/profiles/', endpoint='profilesindex')
def loadContacts():
return send_from_directory(root + 'profiles/', 'index.html')
@static_files_bp.route('/board/<path:path>', endpoint='boardContent')
def boardContent(path):
return send_from_directory(root + 'board/', path)
@static_files_bp.route('/shared/<path:path>', endpoint='sharedContent')
def sharedContent(path):
return send_from_directory(root + 'shared/', path)
@static_files_bp.route('/', endpoint='onionrhome')
def hello():
# ui home
return send_from_directory(root + 'private/', 'index.html')
@static_files_bp.route('/private/<path:path>', endpoint='homedata')
def homedata(path):
return send_from_directory(root + 'private/', path)

6
onionr/httpapi/miscpublicapi/__init__.py Executable file
View File

@ -0,0 +1,6 @@
from . import announce, upload, getblocks, endpoints
announce = announce.handle_announce # endpoint handler for accepting peer announcements
upload = upload.accept_upload # endpoint handler for accepting public uploads
public_block_list = getblocks.get_public_block_list # endpoint handler for getting block lists
public_get_block_data = getblocks.get_block_data # endpoint handler for responding to peers requests for block data

65
onionr/httpapi/miscpublicapi/announce.py Executable file
View File

@ -0,0 +1,65 @@
'''
Onionr - Private P2P Communication
Handle announcements to the public API server
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import base64
from flask import Response
import logger
from etc import onionrvalues
from onionrutils import stringvalidators, bytesconverter
def handle_announce(clientAPI, request):
'''
accept announcement posts, validating POW
clientAPI should be an instance of the clientAPI server running, request is a instance of a flask request
'''
resp = 'failure'
powHash = ''
randomData = ''
newNode = ''
ourAdder = clientAPI._core.hsAddress.encode()
try:
newNode = request.form['node'].encode()
except KeyError:
logger.warn('No node specified for upload')
pass
else:
try:
randomData = request.form['random']
randomData = base64.b64decode(randomData)
except KeyError:
logger.warn('No random data specified for upload')
else:
nodes = newNode + clientAPI._core.hsAddress.encode()
nodes = clientAPI._core._crypto.blake2bHash(nodes)
powHash = clientAPI._core._crypto.blake2bHash(randomData + nodes)
try:
powHash = powHash.decode()
except AttributeError:
pass
if powHash.startswith('0' * onionrvalues.OnionrValues().announce_pow):
newNode = bytesconverter.bytes_to_str(newNode)
if stringvalidators.validate_transport(newNode) and not newNode in clientAPI._core.onionrInst.communicatorInst.newPeers:
clientAPI._core.onionrInst.communicatorInst.newPeers.append(newNode)
resp = 'Success'
else:
logger.warn(newNode.decode() + ' failed to meet POW: ' + powHash)
resp = Response(resp)
if resp == 'failure':
return resp, 406
return resp

80
onionr/httpapi/miscpublicapi/endpoints.py Normal file
View File

@ -0,0 +1,80 @@
'''
Onionr - Private P2P Communication
Misc public API endpoints too small to need their own file and that need access to the public api inst
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
from flask import Response, Blueprint, request, send_from_directory, abort
from . import getblocks, upload, announce
class PublicEndpoints:
def __init__(self, public_api):
client_API = public_api.clientAPI
config = client_API._core.config
public_endpoints_bp = Blueprint('publicendpoints', __name__)
self.public_endpoints_bp = public_endpoints_bp
@public_endpoints_bp.route('/')
def banner():
# Display a bit of information to people who visit a node address in their browser
try:
with open('static-data/index.html', 'r') as html:
resp = Response(html.read(), mimetype='text/html')
except FileNotFoundError:
resp = Response("")
return resp
@public_endpoints_bp.route('/getblocklist')
def get_block_list():
'''Get a list of blocks, optionally filtered by epoch time stamp, excluding those hidden'''
return getblocks.get_public_block_list(client_API, public_api, request)
@public_endpoints_bp.route('/getdata/<name>')
def get_block_data(name):
# Share data for a block if we have it and it isn't hidden
return getblocks.get_block_data(client_API, public_api, name)
@public_endpoints_bp.route('/www/<path:path>')
def www_public(path):
# A way to share files directly over your .onion
if not config.get("www.public.run", True):
abort(403)
return send_from_directory(config.get('www.public.path', 'static-data/www/public/'), path)
@public_endpoints_bp.route('/ping')
def ping():
# Endpoint to test if nodes are up
return Response("pong!")
@public_endpoints_bp.route('/pex')
def peer_exchange():
response = ','.join(client_API._core.listAdders(recent=3600))
if len(response) == 0:
response = ''
return Response(response)
@public_endpoints_bp.route('/announce', methods=['post'])
def accept_announce():
'''Accept announcements with pow token to prevent spam'''
resp = announce.handle_announce(client_API, request)
return resp
@public_endpoints_bp.route('/upload', methods=['post'])
def upload_endpoint():
'''Accept file uploads. In the future this will be done more often than on creation
to speed up block sync
'''
return upload.accept_upload(client_API, request)

52
onionr/httpapi/miscpublicapi/getblocks.py Executable file
View File

@ -0,0 +1,52 @@
'''
Onionr - Private P2P Communication
Public endpoints to get block data and lists
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
from flask import Response, abort
import config
from onionrutils import bytesconverter, stringvalidators
def get_public_block_list(clientAPI, publicAPI, request):
# Provide a list of our blocks, with a date offset
dateAdjust = request.args.get('date')
bList = clientAPI._core.getBlockList(dateRec=dateAdjust)
if clientAPI._core.config.get('general.hide_created_blocks', True):
for b in publicAPI.hideBlocks:
if b in bList:
# Don't share blocks we created if they haven't been *uploaded* yet, makes it harder to find who created a block
bList.remove(b)
return Response('\n'.join(bList))
def get_block_data(clientAPI, publicAPI, data):
'''data is the block hash in hex'''
resp = ''
if stringvalidators.validate_hash(data):
if not clientAPI._core.config.get('general.hide_created_blocks', True) or data not in publicAPI.hideBlocks:
if data in clientAPI._core.getBlockList():
block = clientAPI.getBlockData(data, raw=True)
try:
block = block.encode() # Encode in case data is binary
except AttributeError:
abort(404)
block = bytesconverter.str_to_bytes(block)
resp = block
if len(resp) == 0:
abort(404)
resp = ""
# Has to be octet stream, otherwise binary data fails hash check
return Response(resp, mimetype='application/octet-stream')

43
onionr/httpapi/miscpublicapi/upload.py Executable file
View File

@ -0,0 +1,43 @@
'''
Onionr - Private P2P Communication
Accept block uploads to the public API server
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import sys
from flask import Response, abort
import blockimporter, onionrexceptions, logger
def accept_upload(clientAPI, request):
resp = 'failure'
try:
data = request.form['block']
except KeyError:
logger.warn('No block specified for upload')
pass
else:
if sys.getsizeof(data) < 100000000:
try:
if blockimporter.importBlockFromData(data, clientAPI._core):
resp = 'success'
else:
logger.warn('Error encountered importing uploaded block')
except onionrexceptions.BlacklistedBlock:
logger.debug('uploaded block is blacklisted')
pass
if resp == 'failure':
abort(400)
resp = Response(resp)
return resp

44
onionr/httpapi/onionrsitesapi/__init__.py Normal file
View File

@ -0,0 +1,44 @@
'''
Onionr - Private P2P Communication
view and interact with onionr sites
'''
'''
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
import base64
from flask import Blueprint, Response, request, abort
import core, onionrblockapi, onionrexceptions
from onionrutils import stringvalidators
site_api = Blueprint('siteapi', __name__)
@site_api.route('/site/<name>', endpoint='site')
def site(name):
bHash = name
resp = 'Not Found'
if stringvalidators.validate_hash(bHash):
try:
resp = onionrblockapi.Block(bHash).bcontent
except onionrexceptions.NoDataAvailable:
abort(404)
except TypeError:
pass
try:
resp = base64.b64decode(resp)
except:
pass
if resp == 'Not Found' or not resp:
abort(404)
return Response(resp)

17
onionr/httpapi/simplecache/__init__.py → onionr/httpapi/profilesapi/__init__.py
View File

@ -1,7 +1,7 @@
'''
Onionr - P2P Anonymous Storage Network
Onionr - Private P2P Communication
This file creates http endpoints for friend management
This file creates http endpoints for user profile pages
'''
'''
This program is free software: you can redistribute it and/or modify
@ -19,13 +19,10 @@
'''
import core
from flask import Blueprint, Response, request, abort
from . import profiles
simplecache = Blueprint('simplecache', __name__)
profile_BP = Blueprint('profile_BP', __name__)
@simplecache.route('/get/<key>')
def get_key(key):
return
@simplecache.route('/set/<key>', methods=['POST'])
def set_key(key):
return
@profile_BP.route('/profile/get/<pubkey>', endpoint='profiles')
def get_profile_page(pubkey):
return Response(pubkey)

2
onionr/httpapi/profilesapi/profiles.py Executable file
View File

@ -0,0 +1,2 @@
def get_latest_user_profile(pubkey):
return ''

Some files were not shown because too many files have changed in this diff Show More