Hive Engine nodes were attacked

avatar
(Edited)

We all know that there are a bunch of idiots everywhere. When something is going well, there would always be some negative energy from different sides. From incidents like these, there are always frustrations and learning. For the past few days, the Hive Engine public nodes have been suffering from multiple attacks. I thought it would be better if I give an update on what is happening. Some of my friends asked me what was going on with Hive Engine nodes.

From the looks of it, it was a planned DDoS attack on the public nodes. The list of all available Hive Engine witnesses are available on this page. The website also displays the IP address of the accounts running the witness node. This is quite common and the problem is not because of displaying the IP address. The main problem is that someone decided to attack the Hive Engine nodes with a bad set of queries. Until yesterday, I was not sure if it was an intentional attack or not. But when it happened a couple of times in the last few days, we found out that it was an intentional attack on the public nodes.

In the last few days, many Hive Engine witnesses had their servers running at 100% capacity. You can take a look at the below image to see how all the CPU cores are at 100%. It was a nightmare to run the servers at 100% capacity. It was not a few nodes that were affected. Several nodes including the backup nodes were affected because of this and people had to block the RPC endpoint on the node finally to stop being attacked.

I thought it was an unintentional malformed query that was creating this issue but later we found out that this attack was intentional. If you look at the above two screenshots, the querying happened continuously on multiple nodes at the same time with several IP addresses. It was very hard to track these IP addresses and block them individually.

The initial solution was to introduce rate limiting. Some people introduced a rate limiting on the domain level using Cloudflare, some introduced it on the web server level on nginx and some introduced it directed in the code with the help of express-ratelimiter package. This helped to some extent to block attacks from the same IP address. As you can see above, the attacker was clever and wanted to do the attack from multiple IP addresses. This always kept the CPU percentage at 100%.

The above screenshot was during the peak time when the server was fully loaded. Mostly this was because the Mongo server was very busy trying to serve all the requests from the attacker. The below image was from another powerful server that I had with 12 cores and that also had the same problem.

The server was literally struggling to serve all these requests during the attack. The final solution that worked was to just block the RPC port from the public and enable it only for specific IP addresses. I have associated a domain to some of the nodes that I manage and these domains are behind Cloudflare with a rate-limiting set. On the other nodes, I'm just keeping it private because other services that are running on these hardware are getting affected because of this issue.

Today morning the attacker was again back with another attack. This time the issue was not very big as many nodes have already added rate limiting and some nodes have blocked their RPC port from public access. In the message on the request, the attacker had mentioned: "You Win For Now". This was the funniest thing and this actually confirmed that it was an attack.

Finally, the great thing that happened out of this event is that many great minds worked together on this issue and they have come up with several solutions to handle this problem. Some of these solutions are experimental for now and would soon be incorporated as standard operating procedures. I had a great feeling that "Necessity is the mother of invention" and that way thanks to the attacker. We had to waste a lot of time on this but it was indeed a great learning.


If you like what I'm doing on Hive, you can vote me as a witness with the links below.

Vote @balaz as a Hive Witness

Vote @kanibot as a Hive Engine Witness





0
0
0.000
32 comments
avatar

This post has been manually curated by @bhattg from Indiaunited community. Join us on our Discord Server.

Do you know that you can earn a passive income by delegating to @indiaunited. We share more than 100 % of the curation rewards with the delegators in the form of IUC tokens. HP delegators and IUC token holders also get upto 20% additional vote weight.

Here are some handy links for delegations: 100HP, 250HP, 500HP, 1000HP.

image.png

100% of the rewards from this comment goes to the curator for their manual curation efforts. Please encourage the curator @bhattg by upvoting this comment and support the community by voting the posts made by @indiaunited..

This post received an extra 20.00% vote for delegating HP / holding IUC tokens.

0
0
0.000
avatar

pixresteemer_incognito_angel_mini.png
Bang, I did it again... I just rehived your post!
Week 157 of my contest just started...you can now check the winners of the previous week!
!BEER
9

0
0
0.000
avatar

Those things are a pain, but in the end we will come out stronger :) ... I hope the attacker gets bored and leaves for good!

This post has been manually curated by the VYB curation project

0
0
0.000
avatar

wow this was bad, oh that is the reason why the hive was not working properly, thanks a lot bade bhai for the update. thanks to all the developers and witnesses who came up with the great idea.

!giphy great job

!PIZZA

0
0
0.000
avatar

Having those CPUs at 100% is crazy!! There will always be stupid people who hate progress and like to ruin beautiful things people create. It's just crazy and I just can't wrap my head around it when some bad actors just decide to be wickedly wicked with the knowledge they have on web3. Glad you guys were able to band together to fend off the attacker with timely solutions.

0
0
0.000
avatar

I did not know about this and I do not believe I was affected by it but it bugs me that someone would so something like this. It makes you wonder about the logic behind such an idea.

0
0
0.000
avatar

This is going to make us stronger! Thanks for letting us know what was going on :)

0
0
0.000
avatar

It's bound to happen and will constantly happen. What matters is the speed in which it's fixed and then how to best plug that security issue so it doesn't repeat itself. This is why we bring on smart devs and systems into the witness pool to keep things running.

0
0
0.000
avatar

This is indeed true. There will always be situations like these and the speed in which it is rectified matters a lot.

0
0
0.000
avatar

That's the good part about it. It helps in creating solutions that makes the system more resilient to future attacks. Thanks for the update :)

0
0
0.000
avatar

Yes indeed. The system becomes more strong after an incident like this.

0
0
0.000
avatar

Great to hear that through the struggles new innovation is created.

Keep up the good work!

0
0
0.000
avatar

Yes true. When there is a necessity, a new invention or discovery happens.

0
0
0.000
avatar

I think the fact that it was an intentional attack was a given.
The more successful you get the more then the more often it happens.

0
0
0.000
avatar

Yeah there are a bunch of people on the negative side as well.

0
0
0.000
avatar

I guess that explains why things were a bit funny on all Engine-related activities the past few days.

!discovery 37

0
0
0.000
avatar

I've been at HiveEngine for a long time and the truth is these attacks were a big problem. Thanks for your great work and I'm so glad you fixed it. It is time to prepare for the next attacks


Has sido curado por @visualblock / You've been curated by @visualblock
Bienvenidas delegaciones / Delegations welcome
Encuentra nuestra comunidad aquí / Find our community here
Trail de Curación / Curation Trail

0
0
0.000
avatar
(Edited)

Yeah at least we have some kind of solution right now. But attackers always have new ways to attack. 🙂

0
0
0.000
avatar

Why would someone do that?

0
0
0.000
avatar

Lol yeah there are mad people everywhere. 😃

0
0
0.000
avatar

So much effort to spoil things for others. 😂

Must be a sociopath.

0
0
0.000
avatar

I read that many of them resort to "Ddos" it is because they have a giant botnet in many cases they even infect domestic devices without anyone knowing and attack in the background. The foolish thing about this hacker is not being an active user because to make the attack more effective, she should investigate the flow and times of greatest demand in hive, on the other hand, as bad as these attacks sound, she adds experience because you know what the weak points are in the network :D

0
0
0.000
avatar

There are people who do not create, do not build, and only enjoy trying to damage or destroy the work of others.
They better keep getting stronger, because they will surely keep trying to attack the chain.
They are bad people and very persistent.

0
0
0.000