Interview with Faraz (@Radstakes) from Radstakes.com and @StefanPersson from the Radix Community
@StefanPersson
CAN YOU TELL US A BIT ABOUT YOU AND YOUR TEAM’S BACKGROUND?
Faraz (@Radstakes)
I run Radstakes (https://www.radstakes.com/), which is a validator node on the Radix mainnet based in Wales in the UK. Radstakes consists primarily of me, but I do have a partner who works with me mainly on the monitoring and the administration side. Mainly to make sure that we maximize uptime for our stakers. If there’s ever an issue that I personally can’t attend to, we have an escalation policy that would notify them to step in and take action if needed, if there’s any problems on the node. I’m not a developer or IT professional, I’m actually a mechanical engineer by profession, but I fell into the crypto rabbit hole about eight years ago in 2014 after reading the Satoshi whitepaper. I think the idea of a peer-to-peer currency that was free of government control was the main thing that drew me in initially.
I bought my first Bitcoin for about $500, and regretted selling it ever since of course, but yeah, I just was fascinated by the pace of technological developments in the space over that time. I’ve got a fair bit of experience on the hardware side. I’ve mined proof of work chains such as Etherem, Zcash and other Ethash coins since about 2016, building and running mining rigs from my very small flat at the time, much to the dismay of my wife! Since then I’ve been mainly operating cloud servers for liquidity mining and validating on Radix of course, which I’ve been doing since the very start of betanet.
@StefanPersson
WHAT GOT YOU INTERESTED IN RADIX SPECIFICALLY?
Faraz (@Radstakes)
I think it was around 2018. I just started to get frustrated with the lack of innovation in the crypto space. It seemed like everybody was just copying and pasting Ethereum with very minor tweaks to different parameters and such. This was at the height of the ICO craze as well. I just set out to try and find a real gem that was taking a longer-term view and focusing more on innovation. Scalability and decentralization had always been a concern for me, especially as I’d seen how mining difficulty had escalated so much on Bitcoin and Ethereum. I felt like these networks were starting to converge so only a very small number of miners or pools would have the means to be able to keep up with the pace of that growth.
I could just see things centralizing and that, along with the fact that the fees were just getting astronomical as well, were the main drivers for me to pursue alternatives. I started looking at different networks and projects that were trying to solve the scalability issue, like DAGs such as IOTA, and also layer two solutions like Raiden on Ethereum. Then soon after I came across Radix. I attended the alpha launch event back in 2018 in London, and I was completely blown away. Tempo was the protocol at the time, which I think a lot of Cerberus’s features were inherited from, and the 1.4 million TPS tests that came the following year. That and my impression of Dan and Piers really are what led to my Radix revelation.
@StefanPersson
WHAT MADE YOU DECIDE THAT YOU WANTED TO RUN A VALIDATOR NODE?
Faraz (@Radstakes)
I guess it was probably after the Alpha event in 2018. There was a data capture form to put your name down if you were interested in taking part in betanet testing. I jumped at the chance back then, but it was quite a few years before I actually got the opportunity before the betanet launch. I immediately put my application in to be considered as a betanet validator and fortunately I was accepted. Over the past 4 years I had been looking for an opportunity to participate in and contribute to Radix and node running was the ideal fit for me, especially as I’d not come from a developer background. It was a way of becoming a part of this great community by leveraging some of the experiences and skills I’d gained from mining back in the day.
@StefanPersson
WHAT ARE SOME CHALLENGES THAT ARISE WHEN YOU’RE RUNNING A VALIDATOR?
Faraz (@Radstakes)
I guess the main priority for any validator is maintaining uptime. There’s lots of things that can affect performance: network outages, data corruption, and power failure to name a few. I think having a robust monitoring system being able to react quickly is quite key. Also, the bar is set so high amongst validators, it’s very difficult to differentiate yourself and attract stake from the community. That’s definitely a challenge for someone trying to enter. Centralization risk is a big concern for me, especially when setting out to run a validator because many of us run nodes in the cloud. There are a few that self-host, but predominantly I think most of the nodes are in the cloud. Obviously, we try to avoid the big server providers like AWS, Google and Azure, but recently the Stake Safe (https://www.stakesafe.net/) validators put a great dashboard (https://validators.stakesafe.net/) together that gave some insights into where different validators were hosting their nodes.
It was really quite telling because it identified that about 30% of nodes had hosted their validators at one particular data center, which obviously is a massive risk to the network. If that data center has an outage or if there’s some problem it could be really detrimental to liveness and performance. I’m sorry to say that our node was actually in that data center as well. So, we had to take action and move our validator to a different data center to address that issue and not compound the centralization problem. So, there’s those kinds of technological decisions to be made. I think one of the biggest problems for new validators is just getting into the set. At the moment we’re limited to 100 validators, which means that only the top 100 by total state actually earn rewards and take part in consensus.
Breaking into the set can be a real challenge, although it has been done. Several nodes have managed to get enough support from the community to do it. I think that’s probably the biggest barrier of entry for any new validator until the Xi’an release in 2024 at which point, that limit will be removed.
Speaking of Xi’an, the models around fees and rewards that validators should expect are still being developed by the RDX Works team. So what’s likely to happen as the network grows is we’ll see a range of different validators sets validating subsections of the network. My expectation is that the validators that are working a particular set would still be selected based on the amount of stake that they have, but due to the sharded nature of Radix, even the smallest nodes can participate and help service a small section of the network. They would be rewarded, proportional to the amount of work that they do. That has always been the intention as I understand it, but probably a bit too early for me to speculate as to exactly how this is going to work at Xi’an.
@StefanPersson
THERE HAVE BEEN SOME INTERESTING COLLABORATIONS AND PROMOTIONS IN THE NODE RUNNER COMMUNITY. DOES RADSTAKES.COM HAVE ANYTHING PLANNED?
Faraz (@Radstakes)
I think part of the success of a lot of these validators, especially those that haven’t been around from the very beginning is that they’ve built a lot of their support based on everything else that they’re offering. If we take OCISwap (https://ociswap.com/), for example, they’re building what will probably be the first DEX on Radix, and you’ve got the StakeNordic (https://www.stakenordic.com/) guys who came in quite late and built their offering around NFTs and a marketplace. As for Radstakes, we launched a Radstakes token some months ago (RST), which is currently traded on the Radix planet exchange (https://radixplanet.com/). The idea for our token was to reward loyalty. We airdrop our token periodically to stakers, so the longer you stake with us, the more Radstakes tokens you can get. Holders of the token are then eligible to receive NFTs, and also XRD rewards to boost their earnings from staking alone.
We’ve also got plans to branch out. I’m pleased to admit that I’m learning Scrypto. So in time, hopefully I can expand the Radstakes offering into some decentralized applications after Babylon. I think this is quite important as, like I said earlier, it’s really difficult to set yourself apart as all validators are operating at such a high level of performance. I think there are very few validators that are below 99.9% uptime. For me, I’ve always tried to differentiate myself through trust, transparency and visibility. As a community moderator and ambassador, I’m always engaging with the community and in a space that’s typically characterized by anonymity. I think trust is quite hard to come by.
Stakers are putting a lot of trust in us as validators to do the right thing for the network and to ensure that they continue to earn rewards. At the moment, it’s an easier decision to make choosing a validator because there is currently no risk to stake, but it’s important to remember that slashing is going to be implemented on the network soon, which would put delegated stake at risk if a validator acts maliciously. I’ve always tried to make sure that I’ve got that visibility and trust in the community so that the people can feel confident when they’re entrusting their delegated stake to me.
@StefanPersson
WHAT IS SLASHING?
Faraz (@Radstakes)
Slashing is basically when the protocol penalizes a validator and potentially the stake of their delegators if that validator does something malicious. That would normally be something pretty serious, like trying to commit a double spend or operating your validator key on two nodes at once for example to try and influence consensus. Those kinds of malicious acts would result in some of that stake actually being destroyed. This is something that’s quite important for maintaining the security of a delegated proof-of-stake network such as Radix. It’s something that the team has always been quite keen to implement, but so far we don’t have that in place, but it’s certainly coming. I therefore think having a validator that you can trust and one that’s got some skin in the game is a really important decision for stakers to make.
@StefanPersson
WHAT IS THE TIMELINE FOR IMPLEMENTING SLASHING?
Faraz (@Radstakes)
To be honest, I’m not sure. There are a few node updates that are leading to this. We had our first coordinated fork a few days ago, following that we’ve got a liveliness assurance upgrade that’s planned. I think that’s going to be the precursor to slashing. The liveliness assurance upgrade would basically mean that if a validator performs poorly and has low uptime, they would be removed from consensus. The validator set would basically kick them out and select somebody else to take their place until they can resolve whatever problems they’re experiencing. That kind of liveliness guarantee is kind of the precursor to slashing where the stake of that validator could potentially be at risk, but it’s not clear yet when exactly that might be.
If you want to know about changes like these, you can join the Node Runner Announcement Discord channel.
@StefanPersson
WHAT SHOULD INDIVIDUALS WHO ARE LOOKING TO STAKE THEIR XRD CONSIDER WHEN EVALUATING WHICH VALIDATORS THEY SHOULD CHOOSE?
Faraz (@Radstakes)
I think one of the main things to think about when you’re considering choosing a validator is how much stake do they currently have? In an ideal world, we would have a hundred validators with roughly 1% of the network stake. You’ve got a nice broad range of validators and no single validator is a point of failure that would cause the network performance to suffer. I think choosing a validator that has less than 1 or 2% stake is a good place to start. It’s also important to consider uptime, which is the most important factor for a staker when it comes to rewards. Lots of delegators select their own fee, which you’ll see in Radix ranges from 0% up to 5% or 6%. Fees tend to have a fairly small effect on staking rewards, so the focus should definitely be on uptime.
Monitoring the uptime of validators over a long period of time is a great place to start. Now, there is the official Radix Explorer (https://explorer.radixdlt.com/), which provides a lot of this information, but I tend to use RadixScan (https://www.radixscan.io/). RadixScan has a really great validator table, which not only provides information on who the validators are and links to their websites, but also provides their uptime over the last 30 days, as well as their fees and other really useful information to help people make their decisions.
Also, you want to check if a validator has a website, and that you can contact them easily; that’s really important as well. Most of the validators have telegram and discord channels. Being able to contact them easily and having regular updates from them is really important. I think that those are the key areas that stakers should really consider carefully.
@StefanPersson
HOW IMPORTANT IS DOXING IN YOUR OPINION?
Faraz (@Radstakes)
I think at this point, it’s not critical, particularly as slashing is not yet implemented. Validators have the ability to assign an owner stake, which is a way of a validator to publicly show how much of their own stake they have on their node. You can kind of gauge from that, how much they’re prepared to lose if they ever acted maliciously. When slashing is implemented if you’re choosing a node you probably want to pick someone who’s got some owner stake so that they’re putting their own capital at risk and less likely to do something untoward. That’s quite a factor, but in terms of doxxing in particular, I think it’s also quite important.
If it was looking for a validator, I would tend to pick people that I can communicate with, that I know of, who have a regular presence, and have their contact information readily available.
I’m also KYC’d as a validator with the Radix foundation, which is a requirement to receive the node runner subsidy. I think the majority of the top 100 are also in a similar position. So to summarize, yes I do think doxing is quite an important thing to do in order to maintain trust amongst the community when they’re delegating their stake to you.
However until slashing has been implemented, there’s very little at risk, only staking rewards. If you stake with a validator who isn’t doxxed and they disappear tomorrow, or they suffer some downtime, you’ve always got control of your stake since your stake never leaves your wallet. You can simply un-stake and move your stake to another validator if that ever happens.
@StefanPersson
HOW DOES THE VALIDATOR SUBSIDY WORK?
Faraz (@Radstakes)
The validator subsidy is provided by the Radix Foundation and it pays around $500 US a month to cover validators’ expenses. The idea behind this was basically to level the playing field and allow anybody to run a validator during the early days of the network. Anybody who’s in the top 100 who maintains a high uptime and doesn’t miss too many consensus proposals, (and provided they are KYCed with the foundation) can receive this subsidy to help pay for validator expenses, such as hosting servers, running costs, monitoring systems etc. It’s also a great way of keeping the fee levels down, but also ensuring that validating is accessible to all.
@StefanPersson
WHAT ARE SOME OF THE TECHNOLOGICAL CONCERNS THAT VALIDATORS FACE?
Faraz (@Radstakes)
There are a lot of things that can affect your validators uptime. Some things you can prepare for and mitigate and some things they’re just completely out of your control. If you think of validators on the network a bit like elected officials, we all have to vote periodically on the state of the network and come to a consensus. If your validator node has a network problem and you’re unable to take part in consensus or suffer some latency issues, then that would ultimately mean that you miss those rounds and slow down the pace of the network. The network expects a validator to respond very quickly to those consensus rounds, so if you’ve got poor bandwidth or network connectivity then that would directly affect your uptime. That’s something that you can try and mitigate as best as possible by having a decent uplink to the internet, lots of bandwidth and high quality hardware such as SSDs and substantial RAM.
In addition to network problems, you’ve got issues that you can’t control, such as if your data provider has an outage, or natural disaster that might mean that your node can’t be reached, resulting in your uptime being affected. Ultimately things like power outages, severe weather, and anything else along those lines could affect the reliability of your servers network connection. There’s obviously hardware considerations as well. How much memory you choose to operate with your node, CPU performance, DDoS protection will all play a part.
In the official documentation for node running, there are some recommended specifications for nodes (https://docs.radixdlt.com/main/node-and-gateway/node-setup-introduction.html). Personally, after some extensive testing, I tended to err on the side of caution so my node hardware specs are significantly higher than what Radix would recommend at this stage, mainly because we had issues testing during betanet with the recommended specs that had resulted in a few missed proposals.
We also wanted to future-proof our operations, so our hardware specifications are quite a bit higher than the recommended for this reason. You can find our node specs on our website https://www.radstakes.com/.
However, I should point out that you can successfully run a high uptime validator at the recommended specification. There are several validators that I know of that still run on the recommended specs. For me personally, I wanted to make sure that I was future-proofing for eventualities when the network’s under a lot higher demand and be confident that our hardware will be able to cope with that.
@StefanPersson
WHAT KIND OF REDUNDANCY DO YOU HAVE, AND HOW MUCH OF IT IS AUTOMATED?
Faraz (@Radstakes)
Having a backup node is almost a prerequisite. It’s not mandatory, but it’s really important if you’re serious about ensuring maximum uptime and ensuring that all your stakers receive consistent rewards. It is possible to automate the process of switching to a backup, but it’s quite risky if something goes wrong; you could end up with a situation where your primary and your backup validator are both validating at the same time. At this current point in time, it would result in your missing proposals. That would also be punishable by slashing in the future, so It’s something that we really wanted to try and avoid. I and the majority of the validators that I know tend to have a backup validator that’s synchronized with the network, operating as a full node but not actually doing any validating. If we have an issue with the primary validator, we can stop validating on the primary and then issue a command on the backup node to take control. That switchover takes less than five seconds. We can do that quite easily provided we’ve got access to a terminal and an internet connection.
@StefanPersson
YOU DO THE SWITCHOVER YOURSELF? HOW ARE YOU SET UP FOR WHEN YOU’RE NOT NEAR A COMPUTER?
Faraz (@Radstakes)
Yeah, absolutely. I use Termux (https://play.google.com/store/apps/details?id=com.termux), an Android terminal app, which I can use to access all of my servers wherever I am, so I can respond quite quickly if there’s ever an issue. In addition, my partner also receives alerts from our monitoring system. Any alert that I don’t personally respond to gets escalated to him and then he can take action if I’m not available.
@StefanPersson
THE NODE RUNNER COMMUNITY HAS BEEN QUITE EFFECTIVE AT SELF ORGANIZING. WE SEE THAT WHENEVER A VALIDATOR GOES OVER 3% TOTAL STAKE, THAT VALIDATOR IS ENCOURAGED STOP GROWING BY THE COMMUNITY. CAN YOU SHARE YOUR PERSPECTIVE ON HOW THAT HAS COME ABOUT?
Faraz (@Radstakes)
Yes this has been great to see, a real example of solidarity amongst the community of validators. I guess we all just share this common goal of supporting the network. I’ve met many of the validators in person and connected with the vast majority of them through Discord and Telegram. Whilst we may compete for stake, everybody appreciates that the strength of the network is our collective responsibility. We all support one another, which is really great to see, and it’s become such an amazing community. As well as the discord node runner channel where a lot of the validators hang out, we’ve also got a node runner group on Telegram with most of the top 100 validators. I think for the most part, we’re all pretty well aligned on certain expectations and best practices, such as open communication to the staking community, running backup nodes, and de-registering your node if you’ve got prolonged periods of downtime. De-registering means that you take your node out of consensus, so if there’s a problem it does not drag the performance of the whole network down. These kinds of principles are what inspired the Validator Code of Conduct (https://radixtalk.com/t/validator-code-of-conduct/320), which is something that we published on the RadixTalk (https://radixtalk.com/) forum some months ago as a collaboration between Radstakes (https://www.radstakes.com/) and RadixPool (https://www.radixpool.com/). I think that now has the support of about 25 or 30 validators who voluntarily agreed to follow these principles. One of which is if your validator exceeds 3% stake, you stop accepting delegations. It’s great that other validators have volunteered to agree to these principles, to ensure that we don’t end up with a centralization risk by having a large percentage of stake on any one validator. There was an example fairly recently with Forbole (https://www.forbole.com/) who had crept above the 3% stake mark, so we set out to contact them and encourage them to do the right thing for the strength of the network.
@StefanPersson
YOU’RE THE AUTHOR OF THE VALIDATOR CODE OF CONDUCT ON RADIXTALK. HOW DID THAT COME ABOUT?
Faraz (@Radstakes)
I can’t take all the credit for the code of conduct. I did write it, but it was worked on in collaboration with Stuart, owner of the RadixPool validator. Stuart was operating the RadixTalk forum at the time. We were looking at ways to build the content on that site. One area we wanted to focus on was sensible staking and encouraging best practice amongst validators. We had a few conversations about what principles we’d like to see, and we talked to other validators about their practices for ensuring high availability and uptime.
Soon we realised that a lot of validators had come to operate on many of the same principles. We all appreciate the need to have a backup node, and to have it in a separate data center. If anything happens to your primary server provider, you’ve got a completely independent backup that you can use. So I think it was just born out of that collective idea that we all have a responsibility to the network and it was a great idea to get alignment on those kinds of best practices.
There have been plenty of instances in the past where validators haven’t followed those principles and been caught out. I think the most notable was Viscosity, which was one of the nodes in the top 10. They suffered quite a severe outage which they couldn’t resolve, and clearly had no backup validator, or means to un-register their node. What we ended up with was a severe slowdown to the network which persisted for quite a long time, until finally enough stake was unstaked from their node to a point where the effect on the network was negligible. Eventually, they decided the best course of action was to finally unregister and leave the validator set.
Those kinds of principles really mitigate those risks and are also what inspired the liviness assurance upgrades that are coming in the near future. When this is implemented, The protocol will be able to automatically kick a validator out of consensus if we ever get into that situation, until they become performant again.
@StefanPersson
IS IT COMMON FOR VALIDATORS TO PARTICIPATE IN ANY CASSANDRA RESEARCH?
Faraz (@Radstakes)
Yeah. It’s quite common. I take part in Cassandra testing and I know several of the other validators do as well. It’s a very different ball game to validating on Radix because the protocols are completely different, but it’s a really exciting and interesting thing to do; to be at that cutting edge. It’s a good party atmosphere in the Telegram group (https://t.me/cassandraplayground) as well. Dan basically drops in to suggest a community test and, everybody then frantically spins up validators and tries to support it. It’s a really great thing to be a part of, but it doesn’t necessarily give you any advantages in validating on the mainnet. There’s just so much difference, but still I would definitely encourage anyone to take part in those tests.
Fairly recently Dan tested out the linear scalability of Cassandra. That wasn’t a community test, but I think when we do get to the point where we can community test and you can start to see those throughput numbers skyrocket, it could be really exciting to be a part of.
@StefanPersson
WHAT CHANGES ARE COMING FOR NODE RUNNERS IN THE NEAR FUTURE?
Faraz (@Radstakes)
There’s going to be a lot of changes to the node software. We’ve got several improvements to the protocol coming, some of which I’ve already mentioned, and at Babylon we will have the implementation of Scrypto and the Radix engine, which will all run on ledger. There’s also going to be quite a big change to the software that runs on validator nodes because at the minute, the note is written in Java. The plan is to migrate eventually over to Rust. It will be more consistent with Scrypto and other parts of the network.
As with everything that changes and develops, the RDX Works team do a good job of documenting everything, and there’s a wealth of support in Discord (https://discord.com/channels/417762285172555786/883338066524274695). People can get up to speed fairly quickly and the community is always willing to help people out to make sure that we stay abreast of those changes and updates.
@StefanPersson
IS THERE ANYTHING COMING UP THAT YOUR STAKERS MIGHT BE EXCITED ABOUT?
Faraz (@Radstakes)
Actually yesterday we just completed the next of our Radstakes token airdrops. Anybody who was staking on our validator up until the July 12th snapshot, will have received some Radstakes tokens. Also at the end of the month, we are planning our first Bobcat Society (https://www.bobcatsociety.com/) NFT giveaway. Bobcat Society moved their project from Solana to Radix and I really wanted to offer my support. I got in touch with Lynx and agreed to purchase a bulk order of Bobcat NFTs, which I plan to airdrop to Radstakes stakers. At the end of July, we’ll be dropping 10 Bobcat’s and then we’ll be dropping a further 10 every month for the next 10 months. Anyone who’s holding some Radstakes tokens and staking on our validator will be eligible to take part in the giveaway.
One of the things that we’re also working on at the moment is an uptime guarantee. If we’re ever in a situation where we have prolonged downtime, we want to have a fund of XRD that we can airdrop to our Radstakes token holders to basically compensate them for that downtime. We’re still working out the mechanics of how that’s going to work, but that’s something we hope to be able to offer soon.
@StefanPersson
WHAT OTHER PROJECTS ARE YOU A FAN OF OR ARE PAYING ATTENTION TO?
Faraz (@Radstakes)
I think one group that sticks to mind is CaviarNine. They run validators, but they’re also building some great stuff on Radix. Things like Radit (https://www.radit.io/), their message board on ledger, and they have an AMM swap service as well. I mean, they’ve managed so much even within the confines of Olympia. That makes me really excited about what they might come up with once we get to Babylon and they can leverage the full power of smart contracts and the Radix Engine.
I also attended an event in London recently and heard presentations from Impahla (https://impahla.io/) and also Ideomaker (https://ideomaker.com/). I think they’ve got some great launchpad projects (https://foton.is/#foton-launchpad) and wallets in development so they’re ones to watch.
I’ve mentioned Bobcat Society already, I think that their move to Radix from Solana was a great vote of confidence in Radix. I’m keen to see how they further develop their project.
I’m also paying quite close attention to Radix Planet (https://radixplanet.com/), one of the exchanges that I’ve been involved with. It’s the first order book exchange on Radix. I got to know the founder of the exchange sometime last year and took part in beta testing of that platform. He’s a very talented developer, and I know that he’s also investigating Scrypto at the moment, so we can expect to see some decentralized applications being built at Babylon.
I mean, that’s just in the DeFi space, there’s also people building games and metaverses, and I spoke to someone in discord who’s looking to tokenize real estate. There’s so many great projects out there.
Links
- Website: https://www.radstakes.com/
- Twitter: https://www.twitter.com/Radstakes
