Nginx is one of the most popular web servers in the world and it is responsible for hosting some of the largest and highly loaded sites on the internet. It is more resource-friendly than Apache in most cases and can be used as a web server or reverse proxy.
At Everstake we use a lot of the tools in our services and Nginx is a good fit for many of typical use-cases. In this guide, we’ll show how Nginx can be used to solve common problems that Solana community faces when deploying a validator node.
We took several issues related to the RPC port from Solana’s github: https://github.com/solana-labs/solana/issues and sorted them following the importance order:
- Load balance requests to nodes in cluster
- Stop unauthorized access to RPC by limiting allowed IPs and configuring basic authentication
- Enable HTTPS to secure connections
- Log all requests to improve debug possibilities and use of log-parsers
- Throttle bandwidth or limit requests for snapshot and genesis file downloads
- Functionality to block bad requests
- Get notified about errors on upstream nodes
- Make sure node-operators can customize their setup using available documentation
Prerequisites
Before you begin this guide, you should have:
- Ubuntu 18.04 with Nginx installed
- Deployed Solana TDS validator node, see https://docs.solana.com/tour-de-sol for more details.
We won’t instruct you how to install Nginx on your Ubuntu 18.04 server. There is the decent beginners how-to-guide on DigitalOcean. https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04
Basically, to setup Nginx as a reverse proxy we need Solana RPC (default: 8899) and Websocket (default: RPC+1) ports to bind to another ports.
By specifying next flags in the solana-validator startup command line
— rpc-port 10899
— private-rpc
we change RPC and Websocket to 10899 and 10900 accordingly and disable reachability tests for them. This way we preserve standard ports for Nginx use.
Also, please note that you should not forget to change your firewall settings accordingly.
In the next sections we’ll show excerpts from Nginx config and shortly explain them. Don’t bother rewriting them from screenshots, because we’ll post a link to complete config on github at the end.
1. Load balance requests to nodes in cluster
Load balancing is not the first thing to mention about Nginx, but it is very easy to configure:
This defines a group named solana-proxy which may consist of servers to which you want to proxy requests. Requests are distributed evenly across servers, with server weights taken into consideration. NGINX Open Source supports four load‑balancing methods, and NGINX Plus adds two more methods. Even with freely available Nginx it offers a lot of customization for our needs.
2. Stop unauthorized access to RPC by limiting allowed IP’s and configuring basic authentication
To restrict access we need to create username-password pairs, using a password file creation utility such as apache2-utils and specify the location of generated file in Nginx config. Also we can modify restriction by IP and HTTP authentication with the satisfy directive.
3. Enable HTTPS to secure connections
Nginx can be configured as a load balancer + SSL termination endpoint to distribute incoming traffic across several backend servers in secure way. SSL termination is the process that occurs on the load balancer which handles the SSL encryption/decryption so that traffic between the load balancer and backend servers could be transmitted in HTTP.
This will reduce your SSL management overhead, since the certificate updates can now be managed from the load balancer itself. Consider using Certbot that deploys free Let’s Encrypt certificates or making self-signed ones. CertBot makes cron job for auto renewing them for you.
To make enforcing SSL termination which redirects unsecure http traffic to https make config look like:
Ports left standart to keep things simple to understand. But for Solana TDS RPC you might need to change them this way:
80(unsecure Nginx port) -> 8899 (SSL Nginx endpoint) -> 10899 (backend Solana node)
4. Log all requests to improve debug capabilities and use of log-parsers
NGINX writes information about client requests in the access log right after the request is processed. By default, the information is written to the log in the predefined combined format. To override the default setting, use the log_format directive to change the format of logged messages, as well as the access_log directive to specify the location of the log and its format. Consider using free tool called Nginx Amplify, which needs a little more configuration, but we found it very useful.
We suggest setting variables as follows in the right places as described here:
Using log_format compatible with Amplify https://amplify.nginx.com/docs/guide-metrics-and-metadata.html#additional-nginx-metrics we get extensive info. On top of that, with proper use of log analyzers, such as Goaccess https://goaccess.io/
we can parse them on the fly in terminal or setup a cron job producing analytic info that looks this way:
5. Throttle bandwidth or limit requests for snapshot and genesis file downloads
An attacker can repeatedly download a snapshot or genesis file from a validator that has an open RPC port.
Nginx allows to set different zones for limiting bandwidth, request and connections in a very precise fashion. Below we declare 2 zones based on IP-address to limit requests to 10 req/s, 10 simultaneous connections per IP and throttle download only after 20Mb to 1Mb/s rate:
Premature optimization may be the main problem and first of all we suggest setting higher limits or commenting them out. We used load testing tool https://github.com/rakyll/hey for checking them up.
6. Functionality to block bad requests
Many blockchain projects move REST-like API endpoints that take charge of node operations to different ports and special prefixed URL path that makes easy to protect them. For example, by binding this port to localhost and making restricted firewall rules or simply specifying Nginx prefixes like location ~ ^/(admin|management) to moderate such requests.
But Solana provides simple JSON-RPC which parses http payload in json format, so it makes more troublesome to protect it. However the plugin-based architecture of Nginx makes it possible to deal with it with OpenResty and directive access_by_lua_file. https://github.com/openresty/lua-nginx-module#access_by_lua_file
We found a handy example for unsafe methods blacklisting https://github.com/bedeabza/nginx-jsonrpc-proxy/blob/master/conf/jsonrpc.lua
Taking into account the EOS ecosystem, we say that “The more public endpoints with open RPC — the better” and we encourage node-operators and devs to provide ways to accomplish that kind of true decentralization that blockchain means. We find separating management and public parts of RPC to different ports with ability to bind them on localhost the best solution.
7. Get notified about errors on the upstream nodes
Solana validator tools offer good reporting capabilities for setting up monitoring. But if your infrastructure will be malfunctioning or unreachable you can keep track of Nginx error_log which reports failed upstream servers.
Having alternative monitoring ways is a good thing for maintaining SLA.
8. Make sure node-operators can customize their setup using available documentation
Nginx is a well-documented platform and there are plenty of guides and even online configuration utilities. Though, if you lack some advanced features you can always upgrade to Nginx Plus and preserve your current stack.
Please notice that we don’t show a lot of small tweaks used to manage a decent amount of traffic.
Conclusion
Check out the full config here:
https://gist.github.com/everstake/b0621e6e1db778c0efaac0df1291e6e4
If you want to build a more complex application stack, check out the following documentation.
Thanks for reading!
Links
- https://www.nginx.com/blog/rate-limiting-nginx/
- https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04
- https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04
- https://www.digitalocean.com/community/tutorials/understanding-nginx-server-and-location-block-selection-algorithms
- https://github.com/openresty/openresty
- https://github.com/bedeabza/nginx-jsonrpc-proxy
- https://goaccess.io/
- https://amplify.nginx.com/docs/
- https://www.digitalocean.com/community/tools/nginx
